E XACT S TOCHASTIC N EWTON M ETHOD FOR D EEP L EARNING : THE FEEDFORWARD NETWORKS CASE .
Anonymous authors
Paper under double-blind review
A BSTRACT
The inclusion of second-order information into Deep Learning optimization has drawn consistent interest as a way forward to improve upon gradient descent methods.
Estimating the second-order update is computationally expensive, which drastically limits its usage scope and forces the use of various truncations and approximations.
This work demonstrates that it is possible to solve the Newton direction in the stochastic case exactly.
We consider feedforward networks as a base model, build a second-order Lagrangian which we call Sifrian and provide a closed-form formula for the exact stochastic Newton direction under some monotonicity and regularization conditions.
We propose a convexity correction to escape saddle points and we reconsider the intrinsic stochasticity of the online learning process to improve upon the formulas.
We ﬁnally compare the performance of the developed solution with well-established training methods and show its viability as a training method for Deep Learning.
Optimization in Deep Learning is mainly dominated by ﬁrst-order methods built around the central concept of backpropagation (LeCun et al., 1988).
Second-order methods have exceptional theoretical properties in deterministic optimization (Boyd et al., 2004; Nocedal & Wright, 2006), but such properties do not translate well into the stochastic case (LeCun et al., 2012; Bottou et al., 2018).
First-order methods such as the Stochastic Gradient Descent (SGD) (Robbins & Monro, 1951) are relatively simple and have been adaptively improved for Deep Learning (Duchi et al., 2011; Tieleman & Hinton, 2012; Kingma & Ba, 2014;
Reddi et al., 2019; Yao et al., 2020).
Inherently, secondorder methods might seems inadequate for Deep learning due to increased computational cost, poor clock-wall performance, and the non-convex nature of Deep Learning (LeCun et al., 2012).
Furthermore, Newton method might even reduce the generalization capabilities of training (Wadia et al., 2021; Amari et al., 2020).
Despite these various limitations, substantial effort was deployed to include Hessian information into the optimization process (e.g. Byrd et al., 2011; Sohl-Dickstein et al., 2014; Byrd et al., 2016; Agarwal et al., 2017; Berahas et al., 2019; Anil et al., 2020; Goldfarb et al., 2020; Castera et al., 2021).
Several approaches exist such as the Gauss-Newton method (e.g
Schraudolph, 2002; Botev et al., 2017), diagonal approximation of the Hessian (e.g. Bordes et al., 2009; Schaul et al., 2013), iterative low-rank updates such as BFGS (Broyden, 1970; Fletcher, 1970;
Goldfarb, 1970; Shanno, 1970) (see also, Liu & Nocedal, 1989; Schraudolph et al., 2007; Bollapragada et al., 2018), or Hessian-Free methods which combine the fast Hessian matrix multiplication (Pearlmutter, 1994) and the conjugate gradient algorithm (e.g. Martens, 2010; Martens & Sutskever, 2012; Dauphin et al., 2014).
The use of the Fisher matrix to capture curvature information in the space of distributions, instead of the Hessian, is another approach which yields the natural gradient (Amari, 1998), but suffers from the same computational issues as the Newton method.
Within this context K-FAC method (Martens & Grosse, 2015; Ba et al., 2016a; George et al., 2018) mitigates some of the computational issues of the natural gradient method.
In essence, several drawbacks and ﬂaws limits the adoption of second-order methods as a standard for neural networks training .
In this paper, we propose a novel approach to characterize the Newton update which help us derive an exact closed-form solution for the stochastic Newton method.
Our method requires a suitable regularization of the neural network and strict monotonicity of the activation functions.
We start this paper by introducing useful notations while deriving the well-known backpropagation algorithm for a feedforward network.
Then we introduce a second-order Lagrangian which we call Sifrian, that will serve to characterize the Newton direction.
We derive four types of equations from the Sifrian, and we provide an exact closed-form solution for the Newton direction in the stochastic case.
We further propose a saddle-free version of our method, and add a randomization process to enhance our solution.
In the last part of this paper, we show the applicability of our method for Deep Learning through diverse classiﬁcation tasks using feedforward architectures.
P RELIMINARIES
Deep Learning and neural networks training could be seen as an optimization problem of an expected loss function (cid:96) over a distribution D of labeled samples ( x , d ) .
In general, weights and biases ( W , β ) are the sought-after parameters of the network.
The labeled samples distribution D is often unknown, but a large number of samples (database D ) allows the approximation of the expected loss with an empirical risk:
[Equation]
The loss or cost function (cid:96) is typically a cross entropy function or an l 2 norm of the mismatches between the network outputs and the labels: (cid:96) (cid:0) W , β , (cid:0) x
( p ) , d ( p ) (cid:1)(cid:1) = 1 2 (cid:13) (cid:13) (cid:13) x ( p ) n − d ( p ) (cid:13) (cid:13) (cid:13) 2 2 .
Where n indicates the output layer number.
We present more in detail the architecture of a Feedforward
Neural Network (FNN) hereafter.
1. N OTATIONS AND F EEDFORWARD N EURAL N ETWORKS (FNN)
In this section, we recall the main details of FNNs, which will be used as a standard model for Deep
Learning 1 .
The notation used throughout this paper is similar to notations presented in LeCun et al.
The main equation governing the FNN, a.k.a.
the forward model is the following:
[Equation]
where D is the database, p designs one single sample from the database (e.g. one single image or audio recording), k is the layer number and n is the total number of layers in the network.
The initial input for sample p is x ( p {+) 
+} (e.g. the vectorized input image data).
The state variable x ( p ) is transformed at each layer k through a multiplication by a weight matrix W k and an addition of a bias vector β k .
The activation function F , which is typically a sigmoid or a Rectiﬁed Linear Unit (ReLU), is applied element wise on the resulting activation vector: a ( p ) k
[Equation] serves to introduce non-linearity in the neural network.
1. T HE L AGRANGIAN AND B ACKPROPAGATION
The origin of backpropagation could be traced back to the early 1970s and could be derived by casting Deep Learning as a constrained optimization problem.
This section is similar to LeCun et al.
The main novelty in this section is the addition of a regularization term dependent on the state variable R ( x ) to the cost function (cid:96) : (cid:96) R = (cid:96) + R ( x ) .
The regularization has to verify the admissibility criteria which we deﬁne hereafter:
Deﬁnition 1: A regularization term is admissible if the second derivative of the augmented cost function w.r.t.
to the state variable is separable and non-null, i.e.:
[Equation]
An example of such regularization is the following function:
[Equation]
[Equation] p,k are symmetric positive matrices.
Such a choice might seem atypical since the regularization often concerns the network parameters, mainly the weights or the biases.
This ”admissible” regularization was introduced for the sole purpose of guaranteeing a non-null partial derivative of the cost (cid:96) w.r.t the state variable x ( p ) k . This last property is essential to solve Equ. 16.
We will show later that such a regularization minimizes also the norm of the gradient (Barrett & Dherin, 2020; Smith et al., 2021).
In order to derive the backpropagation algorithm we introduce the following Lagrangian as per the notation of LeCun et al.
[Equation]
The Lagrangian contains the original cost function, the admissible regularization term, and the product of the forward equation with the adjoint state vectors (cid:16) b ( p ) k (cid:17) p,k .
These adjoint vectors are deﬁned for each layer of the network and each sample of the database; however, their values are not ﬁxed yet and will be chosen in a way that simpliﬁes the gradient computation.
If the state variable x veriﬁes the forward equations, then the Lagrangian simpliﬁes to the cost function:
[Equation]
The total derivative of the previous Lagrangian w.r.t.
the weights or biases is the gradient and could be expressed in terms of partial derivatives as follows 2 :
[Equation]
The partial derivatives of the Lagrangian w.r.t to { W k , β k } k =1 ..n are straightforward to compute.
The Fr ´ echet derivatives (cid:16) d x ( p ) m d W k , d x ( p ) m d β k (cid:17) p,k,m are non-trivial to evaluate and the core idea of backpropagation is the selection of the adjoint state variables (cid:16) b ( p ) k (cid:17) p,k which cancels the superﬂuous terms.
Such a simpliﬁcation is achievable if:
[Equation]
Computing the previous partial derivative yields the following backpropagation equation:
[Equation]
1 E is the indicator function, and it is equal to one if the underlying condition E is true and null otherwise.
The nabla operator ∇ F is a diagonal square matrix with an element-wise derivative of its argument (along the diagonal).
The resolution of the backpropagation system could be split into a boundary condition and a backward propagation system.
Further details could be found in LeCun et al.
The particular choice of the adjoint state vectors (cid:16) b ( p ) k (cid:17) p ∈ D,k =1 ..n yields a simple formula for the gradient of the cost function:
[Equation]
The backpropagation algorithm is fundamental for Deep Learning.
It is simple and provides a straightforward solution to an otherwise tedious problem to solve.
Unfortunately, developing an efﬁcient second-order backpropagation method remains elusive.
The current state-of-the-art is based on the use of the R− operator (Pearlmutter, 1994) to compute the product of the Hessian with a given vector without explicitly calculating or storing the Hessian (Martens, 2010; Dauphin et al., 2014;
Agarwal et al., 2017).
In the following section, we provide a different framework for backpropagation which allows the characterization of the second-order Newton direction.
T HE S IFRIAN AND THE E XACT N EWTON D IRECTION
It is legitimate to wonder if the Lagrangian multipliers method could be replicated for higher-order optimization.
We hereafter provide a solution for the second-order case and build an equivalent
Lagrangian which we call the Sifrian.
2. T HE S IFRIAN
The Lagrangian is useful to derive the ﬁrst-order gradient, and it combines the cost function and the forward model multiplied by adjoint variables.
Our approach to extend the Lagrangian consists in creating a function with the forward model, the backward model [Equation] and the deﬁnition of the gradient.
We choose to omit the loss function, as its derivation will generate the gradient again.
Starting from this paradigm, we introduce the Sifrian 3 .
Deﬁnition 2: The Sifrian S of the feed-forward network is deﬁned as follows:
[Equation]
[Equation] The Sifrian includes all the equations generated by the ﬁrst order backpropagation, multiplied by four new adjoint parameters (cid:110) γ ( p ) k , ζ ( p ) k , N k , η k (cid:111) .
While the Lagrangian describes a saddle point (Nocedal & Wright, 2006), the Sifrian, describes an equilibrium and it has a unique null value when all the forward, backward and gradients equations are veriﬁed i.e.:
[Equation]
We call the previous quantity the equilibrated Sifrian, and it is critical to notice its invariance to total derivation w.r.t.
either the weights or biases (i.e., remains null).
Expressing the total derivative of the equilibrated Sifrian using partial derivatives yields the following two equations:
[Equation]
Similar to ﬁrst order backpropagation, we select the new adjoints variables (cid:110) γ [Equation] (cid:111) [Equation]
k
[Equation]
..n to circumvent the calculation of any superﬂuous terms. We make the following choice: [Equation] In this case
[Equation]
after vectorizing the weight gradient G k (and also N k ) [Equation] and concatenating all their values across all layers [Equation]
the Equ. 13 could be cast as the following block matrix system:
[Equation]
The matrix that appears in the previous system ( H ) is the Hessian.
Hence the computation of the Newton direction could be achieved by solving the associated system of four equations in Equ. 14.
The derivation of the four types of Sifrian equations 4 is provided with further details in the Appendix (A.1).
We report hereafter the equations for piece-wise afﬁne activation functions (e
[Equation]
g.,
ReLU or Leaky ReLU).
The general case of C 2 activation function is included in the Appendix (A.1).
[Equation]
Comparison with the R -operator: Characterizing the Hessian effect without explicitly computing its values has been proposed by Pearlmutter (1994), using an operator R which requires the preselection of a speciﬁc direction to study its transformation through the Hessian.
The Sifrian formulation can achieve the same goal by relaxing the two gradient constraints from the Sifrian equations and preselecting the values of { N , η } .
The product requires a forward pass to compute ζ , a backward pass to compute γ .
Finally, the sought-after product is provided by the gradient of the Sifrian w.r.t.
to the weights and biases.
The Sifrian formulation is derived differently from the R -operator. Nevertheless, it can still fulﬁll the same role, and it is more suitable for inversion, which we will conduct hereafter. 2. T HE E XACT S TOCHASTIC N EWTON (ESN)
The resolution of the previous system is intricate in the general case.
However, if we consider a single sample, i.e., the stochastic case and a strictly monotonous activation function, then a closedform solution could be derived. ESN Theorem: If the cost function is admissibly regularized, the activation function F is strictlymonotonous piece-wise afﬁne, the ﬁrst order adjoints (cid:16) b [Equation]
p
[Equation]
k
[Equation]
p,k are non-null and the curvature of the output layer (cid:110) [Equation] (cid:111) is invertible, then the minimal-rank stochastic Newton second-order update has a closed-form solution:
[Equation]
Proof: Under the proposed assumptions, the Sifrian second equation simpliﬁes to γ ( p ) k = − b ( p ) k , which starts a cascade of simpliﬁcations.
The ﬁrst equations becomes 1 k =2 ..n ζ ( p ) k − 1 = 0 .
The last layer of the third equation yields: γ ( p ) n + [Equation] x
[Equation] p ) n [Equation] x
[Equation]
p ) n ζ ( p ) n = 0 .
The remaining two equations become:
[Equation]
The Newton weight is a matrix, and it is determined through its product with the adjoint vector. Inverting the Newton direction is a degenerate problem, but the minimal rank solution is unique, and it is given in Equ. 17.
A major issue still remains: the results of the Newton weight do not extend to the ﬁrst layer (Equ. 18).
To circumvent this shortcoming, we introduce an additional layer before the input, which corresponds to the identity multiplication and a null bias.
This layer remains unchanged throughout the optimization process, and we call it the ”white layer”.
This pre-input extension allows the deﬁnition of an update rule for the ﬁrst layer.
(cid:3)
White layer concept The third equation of the Sifrian allows the characterization of the Newton weight N k only between the 2 nd and the output (n th ) layers.
Such a problem is fundamental and it is legitimate to wonder how is it possible for Hessian-Free methods to work for inversion.
The answer lays within the regularization/damping (Marquardt, 1970) which covers this structural gap.
The important role of the ﬁrst layer for the Newton method has been highlighted recently by Wadia et al.
In our case, and in order to extend the inversion result to the ﬁrst layer, we just introduce an additional layer before the input, which has the same size as the input, and does not bring any modiﬁcation, hence the appellation ”white layer”.
The white layer corresponds to a matrix multiplication with the identity, a null bias, an identity activation function and is unchanged during the training.
[Figure]
Admissible regularization: The ESN solution relies primarily on the existence of the term (cid:18) ∂(cid:96)
[Equation]
(cid:19) which would be null if not for the admissible regularization, whose effect revolves ultimately around minimizing the norm of the state variable (cid:16) x ( p ) k (cid:17) p,k .
Given that the cost function (cid:96) steers the weights/biases to minimize the mismatches, and consequently the adjoints (cid:16) b ( p ) k (cid:17) p,k then the admissible regularization is contributing to the gradient’s norm minimization (Equ. 10), which is a desirable features for generalization purposes (Barrett & Dherin, 2020; Smith et al., 2021).
The ”granular” solution: The admissible regularisation could be removed and this would lead to a ”granular” solution which updates only the bias of the output layer η n .
The granular solution is not practical and it is of no use for the neural network training.
Hypotheses relaxation: Some of the hypotheses of the ESN theorem could be relaxed: notably the invertibility of the output curvature: (cid:110) [Equation] Moore-Penrose inverse could be used instead. The output curvature is often positive deﬁnite and its inversion is simple for e.g. it is the identity for an l 2 cost function.
A DJUSTMENT OF THE E XACT S TOCHASTIC N EWTON (ESN)
The ESN solution is exact and simple to compute; however, it diverges if implemented as it is.
In this section, we ﬁrst address the non-convexity issue and adjust the solution to become a descent direction.
We also explore the stochasticity effect stemming from the samples randomness during the training process and provide an enhancement to the ESN.
For the remainder of this paper we assume also that regularization is a of the same form as Equ. 4.
We start this section by correcting the concave directions of the ESN.
3. E SCAPING THE S ADDLES : A D ESCENT D IRECTION
The minimal rank solution (Equ. 17) has some similarities with the gradient expression (Equ. 10) [Equation] yet it is not a descent direction.
To further illustrate this, we consider the inner product of the Newton and the gradient expressions layer by layer.
For the weights, the product is always positive:
[Equation]
However [Equation] most of the Newton biases corresponds to ascent directions :
[Equation]
The last layer bias nature depends on the values of the regularization and the error.
One of the main advantages of the Sifrian formulation is that we can treat the weights and biases separately and we can exactly pinpoint to the origin of non-convexity: the bias update.
We choose to build our saddle-free variation by just changing the signs of the ascent direction.
This manipulation is straightforward for all layers except the output bias update.
The problematic term corresponds to the ”granular” solution (i.e., no regularization), which overﬁts every sample.
We chose to keep the ”granular” solution as is and switch the sign of the remaining term.
This guarantees that the update is a descent direction 5 .
The saddle-free ESN is the following:
[Equation]
The above convexity correction is signiﬁcantly simpler than the currently existing methods such as the Gauss Newton truncation (e.g. Schraudolph, 2002; Martens, 2010) or the absolute Hessian (Dauphin et al., 2014).
The convexity correction is the ﬁrst step in improving the ESN.
Another intrinsic issue in the above formulation is that we try to optimize a given sample without taking into consideration the yet ”unknown” coming samples.
The essence of stochasticity in still missing.
In the next section, we include such an aspect into the ESN method.
3. R ANDOMIZATION
The ESN solution we have proposed so far is exact but remains deterministic since it focuses on training a single pre-selected sample p .
An intrinsic source of stochasticity in online learning stems from the randomness of the streamed samples.
As such, it is preferable to use random variables to describe the dynamics of ESN.
The two fundamental equations of ESN after discarding the ”granular” solution, adding the white layer are:
[Equation]
Let’s consider Ω = D as the set of possible outcomes of the training samples 6 and randomize the sample index p → p .
Aside from { N , η } all the vectors in the above system become random variables.
We solve the randomized Equ. 22 using the following least-squares criteria:
[Equation]
We changed the ESN into a least-squares problem, which is desirable, given the sizeable wellestablished literature about optimal unbiased estimators for this class of problems (Aitken, 1936;
Kariya & Kurata, 2004).
The minimisation could be done for each layer separately, which reduces the inverse problem dimensionality.
The search for critical points is detailed in the Appendix (A.4)
[Equation]
and yields the following randomized ESN system which involves a Sylvester equation (for N k ) :
[Equation]
[Equation]
We notice that the above system could considerably simplify if the state variables x ( p ) k − 1 is normalized i.e. null mean and identity variance.
This aspect has been extensively explored in Deep Learning through batch, weight, layer and group normalization (Ioffe & Szegedy, 2015; Salimans & Kingma, 2016;
Ba et al., 2016b; Wu & He, 2018).
Normalizing the state variable corresponds to the following transformation:
[Equation]
The Sylvester equation for the N k could be entirely circumvented, by using the proposed VarianceInvariance Theorem described hereafter. Variance-Invariance Theorem: In addition to the hypotheses of the ESN Thereom (Section 2.2).
If ∀ k ∈ [1 ..n ] : Var (cid:16) x (p) k − 1 (cid:17) (cid:31) 0 , then the randomized ESN is invariant to the variances
Var (cid:16) x (p) k − 1 (cid:17) which can be substituted with the identity.
The randomized ESN system becomes: [Equation] [Equation] Proof: The demonstration builds on the invariance of the Newton method to afﬁne transformations and is detailed step by step in the Appendix (A.5)
[Equation] 3. D ISCUSSION Expectations and estimators: We used the expectation E extensively in this paper to derive the theoretical ESN solution.
However, estimators would replace expectation during implementation.
The performance and the computational cost would be directly related to the estimators choice: rank one, moving average, or even batch estimators are all possible with different compromises.
Notes on convergence: Several convergence results for the Newton method exist (e.g Bottou &
Le Cun, 2005; Byrd et al., 2011; Meng et al., 2020).
The ESN convergence ﬁts well within the general framework developed by Sunehag et al.
(2009), under some mild and non-restrictive conditions.
To batch or not to batch: After randomization, ESN could be used in batch mode.
However, there are no guarantees that such a solution solves the ”batch” Sifrian equations.
Therefore, we focus only on using spot or historical data.
R ESULTS
We validate the ESN method for FNNs, using a classiﬁcation task on MNIST dataset (Lecun et al., 1998) [Figure] and also an autoencoder training with the CURVES dataset (Hinton & Salakhutdinov, 2006).
Leaky Relu and He initializer (He et al., 2016; Glorot & Bengio, 2010) are used for all the training.
Further implementation details are available in the Appendix (A.6).
We report the results hereafter:
[Figure]
MNIST classiﬁcation, the architecture is { 784 , 256 , 32 , 10 } .
For CURVES, the autoencoder has the following architecture { 784 , 216 , 64 , 6 , 64 , 216 , 784 } .
Constant learning rates are used 0.001 for ADAM (Kingma & Ba, 2014) and ADAHessian (Yao et al., 2020) and 0.01 for SGD and ESN.
C ONCLUSION
This paper showed that it is possible to derive an exact expression of the stochastic second-order update for Deep Learning via a second-order Lagrangian which we called Sifrian, under some regularization constraints.
We revisited backpropagation using Lagrange multipliers method and developed a new framework to better understand the mechanics of the Newton method at the stochastic level.
This framework yielded unexpected insights on how to correct the non-convexity, the importance of the ﬁrst layer, and how to get a closed-form solution.
We further proceeded to randomize the solution, to better align the training with its stochastic nature and avoid overﬁtting.
We tested the solution using simple training models; and despite the known limitations of the Newton method, the ESN (Exact Stochastic Newton) solution was able to perform as well as adaptive methods in a non-convex setting.
ESN holds great potential but still requires further reﬁnement for batch training, parameters selection and estimators choice.
Generalization of this work to other type of networks is also a research venue which needs further investigation.
R EPRODUCIBILITY S TATEMENT
The results of this paper are reproducible.
Commented Matlab code is included in the supplementary material.
All experiments are run on a workstation with eight cores 2.6 GHz CPU.
A A PPENDIX A. C 2 S IFRIAN EQUATIONS
The inner product used in the Sifrian is either the inner product of vectors or the trace product for matrices, which are equivalent after vectorization:
[Equation]
The derivation is carried through perturbation factorization for e
[Equation]
g.:
[Equation]
We carry the derivation and we report hereafter the full Sifrian equations for a C 2 activation functions:    
[Equation]
A. ESN RESOLUTION FOR C 2 ACTIVATION FUNCTIONS
The stochastic case for C 2 functions has the same type of simpliﬁcations.
Comparing the ﬁrst and second equations:
[Equation]
k (cid:17) = 0 ,
This yields that 1 k =1 ..n − 1 ζ p ( k ) = 0 .
The fourth equation state that (cid:104) N k x ( p ) k − 1 + η k cid:105)
= 0 is always null except for the last layer.
Let’s focus on the last layer.
Three equations allows to characterize it:
[Equation]
If we assume that the C 2 has a non null second derivative everywhere then, the previous three equations yield a speciﬁc value of (cid:104) N n x ( p ) n − 1 + η n {+(cid:105) = 
+} and also ζ ( p ) n :
Except the last layer we have: γ ( p ) k = − b ( p ) k , and a similar system to the piecewise afﬁne case could be derived.
A. E XAMPLE : A SIMPLE REGULARIZATION
In this section we give a concrete simple example to further explain the ESN method.
In the case of an l 2 cost function and regularization deﬁned by:
[Equation]
we have the following solution:
[Equation]
• The previous solution is straightforward to compute; the ﬁrst order backpropagation provides all the elements.
• It is co-linear with the gradient except for the bias of the last layer.
• The adjoints b ( p ) k ) are impacted by choice of the regularization parameter λ .
• The minimal rank Newton solution is not a descent direction.
Most of the biases correspond to an ascent direction.
• The squared norm division is problematic and will lead to divergent solution if the norm of the backpropagated errors is very small i.e. (cid:13)(cid:13)(cid:13) ∇ F (cid:16) a ( p ) k
(cid:17) b ( p ) k (cid:13)(cid:13)(cid:13) 2 2 → 0 .
A sufﬁcient condition to avoid divergence consists in deﬁning layer by layer regularization such as: λ k − 1 ∝ (cid:13)(cid:13)(cid:13) ∇ F (cid:16) a ( p ) k (cid:17) b ( p ) k (cid:13)(cid:13)(cid:13) .
• It is possible to remove the regularization i.e. λ → 0 , this yields a ”granular” solution which states that the second order optimal solution consists in just adjusting the last layer bias with a slope corrected error: (cid:104) ∇ F (cid:16) a ( p ) n (cid:17)(cid:105) − 1 (cid:16) d ( p ) − x ( p ) n (cid:17) .
The granular solution is not suitable for neural network training, as it overﬁts every sample.
ROOF : PROPOSITION
R ANDOMIZED ESN
We start from the randomized least-squares criteria.
[Equation]
The optimum is a critical point i.e. :
[Equation]
Each layer could be optimized separately which is desirable feature that reduce the dimensionality of the problem.
We get the following derivatives:
[Equation]
Integration over Ω yields expectations and two sets of equations:
[Equation]
We simplify the system by right multiplying the second equation with E (cid:104) x (p) k − 1 (cid:105) and substracting,we get:
[Equation]
The above expression corresponds to a Sylvester equation.
The last two equations are the main result of the proposition.
A. I NVARIANCE OF ESN TO NORMALISATION :
In this section we demonstrate that the ESN is invariant to normalisation which would allow for a simpler resolution of the Sylvester Equation.
Such a result is expected for a Newton method, we do the demonstration step by step.
First let’s start by reformulating the forward problem with the normalized state variable s :
[Equation]
The original state variable is an afﬁne transformation of the normalized one:
[Equation]
The matrix Σ k is symmetric and positive.
We assume further that it is deﬁnite
i
[Equation]
e. invertible in this context.
The forward model becomes:
[Equation]
The square-root of the variance and the expectation could be absorbed into the weights/biases:
[Equation]
With the above notations
[Equation]
the forward model becomes:
[Equation]
k (cid:17) .
We use a new adjoint u .
The Lagrangian becomes:
[Equation]
The derivative w.r.t.
s (p) k should be null, which yields the following backpropagation equation.
[Equation]
Given the deﬁnition of W (cid:48)
k +1 .
The adjoint equation is the same as the original, hence u = b , and the ﬁrst order adjoint is unmodiﬁed.
The gradient is slightly modiﬁed:
[Equation]
Let’s move to the Sifrian:
[Equation]
The Sifrian equations are:
[Equation]
We follow the same steps of the ESN Theorem demonstration, add the white layer and we remove the granular solution.
We ﬁnally get the following stochastic system:
[Equation]
The randomization yields, 
[Equation]
The s is normalized hence:
[Equation]
The randomized, ESN with the normalized state variable has the following solution:
[Equation]
We revert to the original variables from Equ. A.5, add convexity correction and we get the following solution:
[Equation]
A. I MPLEMENTATION DETAILS :
In this section we specify technical details of our implementation.
First [Equation] we remove the ”granular” terms from our solution
[Equation]
since they do overﬁt each sample by construction.
[Equation]
ESN is solved for each layer independently, which means that the inverted matrices are a scale smaller than the empirical Fisher matrix for e.g..
To further illustrate this point, let’s consider, for e.g., the feedforward network with layers sizes [784 , 256 , 32 , 10] (MNIST classiﬁcation): the empirical Fisher matrix would have a size of 210298x210298, while the biggest matrix inversion for ESN would be only 256x256.
In practice, the computation requires initially two expectations
[Equation]
which could be estimated using exponentially decaying moving averages (Similar to ADAM).
The product with the non-centered precision matrix
E (cid:104) I + y [Equation] p
[Equation] k y [Equation] − 1 , could be approximated in several ways.
We list hereafter at least three methods:
• Direct method: the non-centered covariance matrix could be estimated using an exponentially decaying moving average and inverted directly given the small sizes.
• Recursive Least-Squares (RLS): A sliding window could be used to update recursively the estimation of N ( k ) using Sherman-Morrison formula with an initialization of the precision matrix at I .
Such a method is a classic of online learning and is similar to Kalman ﬁltering.
• Rank one method: instead of involving the historical data, the simplest/crudest estimation of the covariance matrix is based on the spot/instantaneous information.
The precision matrix could be derived using Sherman-Morrison formula (applied once):
[Equation]
Probability distribution/weights: It is common to use weights in the context of least-squares regression.
For Deep Learning, we could increase the importance of samples that are not well assimilated into the network, i.e., high errors(or vice-versa).
This is possible for e.g. by choosing probabilities/weights proportional to errors’ norms: w ( p ) ∼ (cid:13)(cid:13)(cid:13) x ( p ) n − d ( p ) (cid:13)(cid:13)(cid:13) 2 2 . The ESN regression equations become:
[Equation]