S TABILIZING G RADIENTS FOR D EEP N EURAL N ETWORKS VIA E FFICIENT SVD P ARAMETERIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Vanishing and exploding gradients are two of the main obstacles in training deep neural networks, especially in capturing long range dependencies in recurrent neural networks (RNNs).
In this paper, we present an efﬁcient parametrization of the transition matrix of an RNN that allows us to stabilize the gradients that arise in its training.
Speciﬁcally, we parameterize the transition matrix by its singular value decomposition (SVD), which allows us to explicitly track and control its singular values.
We attain efﬁciency by using tools that are common in numerical linear algebra, namely Householder reﬂectors for representing the orthogonal matrices that arise in the SVD.
By explicitly controlling the singular values, our proposed svdRNN method allows us to easily solve the exploding gradient problem and we observe that it empirically solves the vanishing gradient issue to a large extent.
We note that the SVD parameterization can be used for any rectangular weight matrix, hence it can be easily extended to any deep neural network, such as a multi-layer perceptron.
Theoretically, we demonstrate that our parameterization does not lose any expressive power, and show how it potentially makes the optimization process easier.
Our extensive experimental results also demonstrate that the proposed framework converges faster, and has good generalization, especially in capturing long range dependencies, as shown on the synthetic addition and copying tasks.
I NTRODUCTION
Deep neural networks have achieved great success in various ﬁelds, including computer vision, speech recognition, natural language processing, etc.
Despite their tremendous capacity to ﬁt complex functions, optimizing deep neural networks remains a contemporary challenge.
Two main obstacles are vanishing and exploding gradients, that become particularly problematic in Recurrent
Neural Networks (RNNs) since the transition matrix is identical at each layer, and any slight change to it is ampliﬁed through recurrent layers (Bengio et al.
Several methods have been proposed to solve the issue, for example, Long Short Term Memory (LSTM) (Hochreiter & Schmidhuber (1997)) and residual networks (He et al.
Another recently proposed class of methods is designed to enforce orthogonality of the square transition matrices, such as unitary and orthogonal RNNs (oRNN) (Arjovsky et al.
; Mhammedi et al.
However, while these methods solve the exploding gradient problem, they limit the expressivity of the network.
In this paper, we present an efﬁcient parametrization of weight matrices that arise in a deep neural network, thus allowing us to stabilize the gradients that arise in its training, while retaining the desired expressive power of the network.
In more detail we make the following contributions:
• We propose a method to parameterize weight matrices through their singular value decomposition (SVD).
Inspired by (Mhammedi et al.
, we attain efﬁciency by using tools that are common in numerical linear algebra, namely Householder reﬂectors for representing the orthogonal matrices that arise in the SVD.
The SVD parametrization allows us to retain the desired expressive power of the network, while enabling us to explicitly track and control singular values.
• We apply our SVD parameterization to recurrent neural networks to exert spectral constraints on the RNN transition matrix.
Our proposed svdRNN method enjoys similar space and time complexity as the vanilla RNN.
We empirically verify the superiority of svdRNN over RNN/oRNN, in some case even LSTMs, over an exhaustive collection of time series classiﬁcation tasks and the synthetic addition and copying tasks, especially when the network depth is large.
• Theoretically, we show how our proposed SVD parametrization can make the optimization process easier.
Speciﬁcally, under a simple setting, we show that there are no spurious local minimum for the linear svdRNN in the population risk.
• Our parameterization is general enough to eliminate the gradient vanishing/exploding problem not only in RNNs, but also in various deep networks.
We illustrate this by applying SVD parametrization to problems with non-square weight matrices, speciﬁcally multi-layer perceptrons (MLPs) and residual networks.
We now present the outline of our paper.
In Section 2, we discuss related work, while in Sectionwe introduce our SVD parametrization and demonstrate how it spans the whole parameter space and does not limit expressivity.
In Section 4 we propose the svdRNN model that is able to efﬁciently control and track the singular values of the transition matrices, and we extend our parameterization to non-square weight matrices and apply it to MLPs in Section 5.
Section 6 provides the optimization landscape of svdRNN by showing that linear svdRNN has no spurious local minimum.
Experimental results on MNIST and a popular time series archive are present in Section 7.
Finally, we present our conclusions and future work in Section 8.
R ELATED W ORK
Numerous approaches have been proposed to address the vanishing and exploding gradient problem.
Long short-term memory (LSTM) (Hochreiter & Schmidhuber (1997)) attempts to address the vanishing gradient problem by adding additional memory gates.
Residual networks (He et al.
(2016)) pass the original input directly to the next layer in addition to the original layer output.
Mikolov (2012) performs gradient clipping, while Pascanu et al.
(2013) applies spectral regularization to the weight matrices.
Other approaches include introducing L 1 or L 2 penalization on successive gradient norm pairs in back propagation (Pascanu et al.
Recently the idea of restricting transition matrices to be orthogonal has drawn some attention.
Le et al.
(2015) proposed initializing recurrent transition matrices to be identity or orthogonal (IRNN).
This strategy shows better performance when compared to vanilla RNN and LSTM.
However, there is no guarantee that the transition matrix is close to orthogonal after a few iterations.
The unitary RNN (uRNN) algorithm proposed in Arjovsky et al.
(2016) parameterizes the transition matrix with reﬂection, diagonal and Fourier transform matrices.
By construction, uRNN ensures that the transition matrix is unitary at all times.
Although this algorithm performs well on several small tasks, Wisdom et al.
(2016) showed that uRNN only covers a subset of possible unitary matrices and thus detracts from the expressive power of RNN.
An improvement over uRNN, the orthogonal RNN (oRNN), was proposed by Mhammedi et al.
oRNN uses products of Householder reﬂectors to represent an orthogonal transition matrix, which is rich enough to span the entire space of orthogonal matrices.
Meanwhile, Vorontsov et al.
(2017) empirically demonstrate that the strong constraint of orthogonality limits the model’s expressivity, thereby hindering its performance.
Therefore, they parameterize the transition matrix by its SVD, W = U Σ V (cid:62) (factorized RNN) and restrict Σ to be in a range close to 1; however, the orthogonal matrices U and V are updated by geodesic gradient descent using the Cayley transform, thereby resulting in time complexity cubic in the number of hidden nodes which is prohibitive for large scale problems.
Motivated by the shortcomings of the above methods, our work in this paper attempts to answer the following questions: Is there an efﬁcient way to solve the gradient vanishing/exploding problem without hurting expressive power?
As brought to wide notice in He et al. (2016), deep neural networks should be able to preserve features that are already good.
Hardt & Ma (2016) consolidate this point by showing that deep linear residual networks have no spurious local optima.
In our work, we broaden this concept and bring it to the area of recurrent neural networks, showing that each layer is not necessarily near identity, but being close to orthogonality sufﬁces to get a similar result.
Generalization is a major concern in training deep neural networks.
Bartlett et al.
(2017) provide a generalization bound for neural networks by a spectral Lipschitz constant, namely the product of spectral norm of each layer.
Thus, our scheme of restricting the spectral norm of weight matrices reduces generalization error in the setting of Bartlett et al.
As supported by the analysis in
Cisse et al.
(2017), since our SVD parametrization allows us to develop an efﬁcient way to constrain the weight matrix to be a tight frame (Tropp et al.
, we consequently are able to reduce the sensitivity of the network to adversarial examples.
SVD PARAMETERIZATION
The SVD of the transition matrix W ∈ R n × n of an RNN is given by W = U Σ V T , where Σ is the diagonal matrix of singular values, and U, V ∈ R n × n are orthogonal matrices, i.e., U T U = UU T =
I and V T V = V V T
=
I (Trefethen & Bau III (1997)).
During the training of an RNN, our proposal is to maintain the transition matrix in its SVD form.
However, in order to do so efﬁciently, we need to maintain the orthogonal matrices U and V in compact form, so that they can be easily updated by forward and backward propagation.
In order to do so, as in Mhammedi et al.
(2017), we use a tool that is commonly used in numerical linear algebra, namely Householder reﬂectors (which, for example, are used in computing the QR decomposition of a matrix).
Given a vector u ∈ R k , k ≤ n , the n × n Householder reﬂector H nk ( u ) is deﬁned as:
[Equation]
The Householder reﬂector is clearly a symmetric matrix, and it can be shown that it is orthogonal, i.e., H
= I (Householder (1958)).
Further, when u (cid:54) = 0 , it has n − 1 eigenvalues that are 1, and one eigenvalue which is − 1 (hence the name that it is a reﬂector) .
In practice, to store a Householder reﬂector, we only need to store u ∈ R k rather than the full matrix.
Given a series of vectors { u i } ni = k where u k ∈ R k , we deﬁne the map:
[Equation]
where the right hand side is a product of Householder reﬂectors, yielding an orthogonal matrix (to make the notation less cumbersome, we remove the superscript from H nk for the rest of this section).
Theorem 1.
The image of M 1 is the set of all n × n orthogonal matrices.
The proof of Theorem 1 is an easy extension of the Householder QR factorization Theorem, and is presented in Appendix A. Although we cannot express all n × n matrices with M k , any W ∈ R n × n can be expressed as the product of two orthogonal matrices
U, V and a diagonal matrix Σ , i.e. by its SVD: W = U Σ V (cid:62) .
Given σ ∈ R n and { u i } ni = k 1 , { v i } ni = k 2 with u i , v i ∈
R i , we ﬁnally deﬁne our proposed SVD parametrization:
[Equation]
Theorem 2.
The image of M 1 , 1 is the set of n × n real matrices.
i
[Equation]
The proof of Theorem 2 is based on the singular value decomposition and Theorem 1, and is presented in Appendix A.
The astute reader might note that M 1 , 1 seemingly maps an input space of n 2 + 2 n dimensions to a space of n 2 dimensions; however, since H nk ( u k ) is invariant to the norm of u k , the input space also has exactly n 2 dimensions.
Although Theorems 1 and 2 are simple extensions of well known linear algebra results, they ensure that our parameterization has the ability to represent any matrix and so the full expressive power of the RNN is retained.
Theorem 3.
The image of M k 1 ,k 2 includes the set of all orthogonal n × n matrices if k 1 + k 2 ≤ n +2 .
Theorem 3 indicates that if the total number of reﬂectors is greater than n : ( n − k 1 +1)+( n − k 2 + 1) ≥ n , then the parameterization covers all orthogonal matrices.
Note that when ﬁxing σ = 1 ,
M k 1 ,k 2 ( { u i } ni = k 1 , { v i } ni = k 2 , 1 ) ∈
O ( n ) , where O ( n ) is the set of n × n orthogonal matrices.
Thus when k 1 + k 2 ≤ n + 2 , we have O ( n ) =
M k 1 ,k 2 (cid:2)
R k 1 × ...
× R n × R k 2 × ...
× R n × 1 (cid:3) .
SVD -RNN
In this section, we apply our SVD parameterization to RNNs and describe the resulting svdRNN algorithm in detail.
Given a hidden state vector from the previous step h ( t − 1) ∈ R n and input x ( t − 1) ∈ R n i , RNN computes the next hidden state h ( t ) and output vector o ( t ) ∈ R n o as:
[Equation]
[Equation]
In svdRNN we parametrize the transition matrix W ∈ R n × n using m 1 + m 2 Householder reﬂectors as:
[Equation]
This parameterization gives us several advantages over the regular RNN.
First, we can select the number of reﬂectors m 1 and m 2 to balance expressive power versus time and space complexity.
By
Theorem 2, the choice m 1 = m 2 = n gives us the same expressive power as vanilla RNN.
Notice oRNN could be considered a special case of our parametrization, since when we set m 1 + m 2 ≥ n and σ = 1 , we can represent all orthogonal matrices, as proven by Theorem 3.
Most importantly, we are able to explicitly control the singular values of the transition matrix.
In most cases, we want to constrain the singular values to be within a small interval near 1.
The most intuitive method is to clip the singular values that are out of range.
Another approach would be to initialize all singular values to 1, and add a penalty term (cid:107) σ − 1 (cid:107) 2 to the objective function.
Here, we have applied another parameterization of σ proposed in Vorontsov et al.
[Equation]
[Equation]
where f is the sigmoid function and ˆ σ i is updated from u i , v i via stochastic gradient descent.
The above allows us to constrain σ i to be within [ σ ∗ − r, σ ∗ + r ] .
In practice, σ ∗ is usually set to 1 and r (cid:28) 1 .
Note that we are not incurring more computation cost or memory for the parameterization.
For regular RNN, the number of parameters is ( n o + n i + n + 1) n , while for svdRNN it is ( n o + n i + m 1 + m 2 + 2) n − m 21 + m 22 − m 1 − m 2.
In the extreme case where m 1 = m 2 = n , it becomes ( n o + n i + n + 3) n .
Later we will show that the computational cost of svdRNN is also of the same order as RNN in the worst case.
4. F ORWARD / BACKWARD PROPAGATION
In forward propagation, we need to iteratively evaluate h ( t ) from t = 0 to L using (4) .
The only different aspect from a regular RNN in the forward propagation is the computation of Wh ( t − 1) .
Note that in svdRNN, W is expressed as product of m 1 + m 2 Householder matrices and a diagonal matrix.
Thus Wh ( t − 1) can be computed iteratively using ( m 1 + m 2 ) inner products and vector additions.
Denoting ˆ
u
k
=
(cid:0)
n − k u k (cid:1) , we have:
[Equation]
[Equation]
Thus, the total cost of computing Wh ( t − 1) is O (( m 1 + m 2 ) n ) ﬂoating point operations (ﬂops).
Detailed analysis can be found in Section 4.2.
Let L ( { u i } , { v i } , σ, M, Y, b ) be the loss or objective function, C ( t ) =
Wh ( t ) , ˆΣ = diag (ˆ σ ) .
Given ∂L∂C ( t ) , we deﬁne:
[Equation]
[Equation]
Back propagation for svdRNN requires ∂C ( t ) ∂u ( t ) k , ∂C ( t ) ∂v ( t ) k , ∂C ( t )
∂ ˆΣ ( t ) and ∂C ( t ) ∂h ( t − 1) .
These partial gradients can also be computed iteratively by computing the gradient of each Householder matrix at a time.
We drop the superscript ( t ) now for ease of exposition.
Given ˆ h = H k
( u k ) h and g
=
∂L∂ ˆ h , we have ∂L
[Equation] [Equation]
Details of forward and backward propagation can be found in Appendix (B).
One thing worth noticing is that the oRNN method in Mhammedi et al.
(2017) actually omitted the last term in (15) by assuming that (cid:107) u k (cid:107) are ﬁxed.
Although the scaling of u k in the Householder transform does not affect the transform itself, it does produce different gradient update for u k even if it is scaled to norm 1 afterwards.
4. C OMPLEXITY A NALYSIS
Table 1 gives the time complexity of various algorithms.
Hprod and Hgrad are deﬁned in Algorithm 2 3 (see Appendix (B)).
Algorithm 2 needs 6 k ﬂops, while Algorithm 3 uses (3 n + 10 k ) ﬂops.
Since (cid:107) u k (cid:107) 2 only needs to be computed once per iteration, we can further decrease the ﬂops to 4 k and (3 n + 8 k ) .
Also, in back propagation we can reuse α in forward propagation to save 2 k ﬂops.
[Table]
E XTENDING SVD P ARAMETERIZATION TO G ENERAL W EIGHT M ATRICES
In this section, we extend the parameterization to non-square matrices and use Multi-Layer Perceptrons(MLP) as an example to illustrate its application to general deep networks.
For any weight matrix W ∈ R m × n (without loss of generality m ≤ n ), its reduced SVD can be written as:
[Equation]
where U ∈ R m × m , Σ ∈ diag ( R m ) , V L ∈ R n × m
There exist u n , ..., u k 1 and v n , ..., v k 2 s.t. U =
H mm ( u m ) ...
H mk 1 ( u k 1 ) , V = H nn ( v n ) ...
H nk 2 ( v k 2 ) , where k 1 ∈ [ m ] , k 2 ∈ [ n ] .
Thus we can extend the
SVD parameterization for any non-square matrix:
[Equation]
where ˆΣ =
( diag ( σ ) | 0) if m < n and ( diag ( σ ) | 0) (cid:62) otherwise.
Next we show that we only need 2 min( m, n ) reﬂectors (rather than m + n ) to parametrize any m × n matrix.
By the deﬁnition of
H nk , we have the following lemma:
Lemma 1.
Given { v i } ni =1 , deﬁne V ( k )
=
H nn ( v n ) ...
H nk ( v k ) for k ∈ [ n ] .
We have:
[Equation]
Here V ∗ ,i indicates the i th column of matrix V .
According to Lemma 1, we only need at most ﬁrst m Householder vectors to express V L , which results in the following Theorem:
Theorem 4.
If m ≤ n , the image of M m,n 1 ,n − m +1 is the set of all m × n matrices; else the image of
M m,nn − m +1 , 1 is the set of all m × n matrices.
Similarly if we constrain u i , v i to have unit length, the input space dimensions of M m,n 1 ,n − m +1 and
M m,nm − n +1 , 1 are both mn , which matches the output dimension.
Thus we extend Theorem 2 to the non-square case, which enables us to apply SVD parameterization to not only the RNN transition matrix, but also to general weight matrices in various deep learning models.
For example, the
Multilayer perceptron (MLP) model is a class of feedforward neural network with fully connected layers:
[Equation]
Here h ( t ) ∈ R n t , h ( t − 1) ∈ R n t − 1 and W ( t ) ∈ R n t × n t − 1 .
Applying SVD parameterization to W ( t ) say n t < n t − 1 , we have:
[Equation]
We can use the same forward/backward propagation algorithm as described in Algorithm 1.
Besides
RNN and MLP, SVD parameterization method also applies to more advanced frameworks, such as
Residual networks and LSTM, which we will not describe in detail here.
T HEORETICAL A NALYSIS
Since we can control and upper bound the singular values of the transition matrix in svdRNN, we can clearly eliminate the exploding gradient problem.
In this section, we now analytically illustrate the advantages of svdRNN with lower-bounded singular values from the optimization perspective.
For the theoretical analysis in this section, we will limit ourselves to a linear recurrent neural network, i.e., an RNN without any activation.
6. R EPRESENTATIONS OF RNN WITHOUT ACTIVATION
Linear recurrent neural network.
For simplicity, we follow a setting similar to Hardt & Ma (2016).
For compact presentation, we stack the input data as X ∈ R n × t , where X = (cid:0) x (0)
| x (1) | · · · | x ( t − 1) (cid:1) , and transition weights as W ∈ R n × nt where W = (cid:0) W | W 2 | · · · | W t (cid:1) .
Then we can simplify the output as:
[Equation]
By absorbing M and b in each data x ( t ) and assuming h (0) =
0 , we further simplify the output as:
[Equation]
Suppose the input data X ∼ D , and assume its underlying relation to the output is y
=
A vec ( X )+ η , where A ∈ R n × nt and residue η ∈ R n satisﬁes E X∼D
[ η |X ]
= 0 .
We consider the individual loss:
[Equation]
Claim 1.
With linear recurrent neural networks, the population risk
[Equation]
2 F + C, where Σ = E X∼D
[ vec ( X ) vec ( X ) (cid:62) ] , and C = E [ (cid:107) η (cid:107) 22 ] .
Meanwhile
[Equation]
6. A LL CRITICAL POINTS ARE GLOBAL MINIMUM
Theorem 5.
With linear recurrent neural networks, if transition matrix W satisﬁes σ min ( W ) ≥ e > 0 , all critical points in the population risk are global minimum.
Proof.
Write Y W − A as ( E 1 | E 2 | · · · | E t ) , where each E i ∈ R d × d .
By Claim 1,
[Equation]
Therefore when ∇ W R [ W ] = 0 sufﬁces R ( W ) =
R ∗ , meaning W reaches the global minimum.
Theorem 5 potentially explains why our system is easier to optimize, since with our scheme of SVD parametrization, we have the following corollary.
Corollary 1.
With the update rule in (8) , linear svdRNNs have no spurious local minimum.
While the above analysis lends further credence to our observed experimental results, we leave it to future work to perform a similar analysis in the presence of non-linear activation functions.
E XPERIMENTAL R ESULTS
In this section, we provide empirical evidence that shows the advantages of SVD parameterization in both RNNs and MLPs.
For RNN models, we compare our svdRNN algorithm with (vanilla) RNN,
IRNN(Le et al.
, oRNN(Mhammedi et al.
and LSTM (Hochreiter & Schmidhuber (1997)).
The transition matrix in IRNN is initialized to be orthogonal while other matrices are initialized by sampling from a Gaussian distribution.
For MLP models, we implemented vanilla
MLP, Residual Network (ResNet)(He et al. (2016)) and used SVD parameterization for both of them.
We used a residual block of two layers in ResNet.
In most cases leaky Relu is used as activation function, except for LSTM, where leaky Relu will drastically harm the performance.
To train these models, we applied Adam optimizer with stochastic gradient descent (Kingma & Ba (2014)).
These models are implemented with Theano (Al-Rfou et al.
7. T IME S ERIES C LASSIFICATION
In this experiment, we focus on the time series classiﬁcation problem, where time series are fed into RNN sequentially, which then tries to predict the right class upon receiving the sequence end (H¨usken & Stagge (2003)).
The dataset we choose is the largest public collection of classlabeled time-series with widely varying length, namely, the UCR time-series collection from Chen et al.
We present the test accuracy on 20 datasets with RNN, LSTM, oRNN and svdRNN in Table 3(Appendix C) and Figure 1.
In all experiments, we used hidden dimension n h = 32 , and chose total number of reﬂectors for oRNN and svdRNN to be m = 16 (for svdRNN m 1 = m 2 = 8 ).
We choose proper depth t as well as input size n i .
Given sequence length L , since tn i =
L , we choose n i to be the maximum divisor of L that satisﬁes depth ≤ √ L .
To have a fair comparison
[Figure]
of how the proposed principle itself inﬂuences the training procedure, we did not use dropout in any of these models.
As illustrated in the optimization process in Figure 1, this resulted in some overﬁtting (see (a) CBF), but on the other hand it shows that svdRNN is able to prevent overﬁtting.
This supports our claim that since generalization is bounded by the spectral norm of the weights Bartlett et al.
(2017), svdRNN will potentially generalize better than other schemes.
This phenomenon is more drastic when the depth is large (e.g. ArrowHead(251 layers) and FaceAll(131 layers)), since regular RNN, and even LSTM, have no control over the spectral norms.
Also note that there are substantially fewer parameters in oRNN and svdRNN as compared to LSTM.
7. MNIST
In this experiment, we compare different models on the MNIST image dataset.
The dataset was split into a training set of 60000 instances and a test set of 10000 instances.
The 28 × 28 MNIST pixels are ﬂattened into a vector and then traversed by the RNN models.
Table 2 shows accuracy scores across multiple We tested different models with different network depth as well as width.
Figure 2(a)(b) shows the test accuracy on networks with 28 and 112 layers (20 and 128 hidden dimensions) respectively.
It can be seen that the svdRNN algorithms have the best performance and the choice of r (the amount that singular values are allowed to deviate from 1) does not have much inﬂuence on the ﬁnal precision.
Also we explored the effect of different spectral constraints and explicitly tracked the spectral margin ( max i | σ i − 1 | ) of the transition matrix.
Intuitively, the inﬂuence of large spectral margin should increase as the network becomes deeper.
Figure 2(d) shows the spectral margin of different RNN models.
Although IRNN has small spectral margin at ﬁrst few iterations, it quickly deviates from orthogonal and cannot match the performance of oRNN and svdRNN.
Figure 2(e) shows the magnitude of ﬁrst layer gradient (cid:107) ∂L∂h
(0) (cid:107) 2 .
RNN suffers from vanishing gradient at ﬁrst 50k iterations while oRNN and svdRNN are much more stable.
Note that
LSTM can perform relatively well even though it has exploding gradient in the ﬁrst layer.
We also tested RNN and svdRNN with different amount of non-linearity, as shown in Figure 2(c).
This is achieved by adjusting the leak parameter in leaky Relu :
f ( x ) = max( leak · x, x ) .
With leak = 1 . 0 , it reduces to the identity map and when leak =
we are at the original Relu function.
From the ﬁgures, we show that svdRNN is resistant to different amount of non-linearity, namely converge faster and achieve higher accuracy invariant to the amount of the leak factor.
To explore the reason underneath, we illustrate the gradient in Figure 2(f), and ﬁnd out svdRNN could eliminate the gradient vanishing problem on all circumstances, while RNN suffers from gradient vanishing when non-linearity is higher.
[Figure]
(f)
[Figure]
[Table]
For the MLP models, each instance is ﬂattened to a vector of length 784 and fed to the input layer.
After the input layer there are 40 layers with hidden dimension 32 (Figure 3(a)) or 30 to 100 layers with hidden dimension 128 (Figure 3(b)).
On a 40-layer network, svdMLP and svdResNet achieve similar performance as ResNet while MLP’s convergence is slower.
However, when the network is deeper, both MLP and ResNet start to fail.
With n h = 128 , MLP is not able to function with L >and ResNet with L > 70 .
On the other hand, the SVD based methods are resilient to increasing depth and thus achieve higher precision.
[Figure]
C ONCLUSIONS
In this paper, we have proposed an efﬁcient SVD parametrization of various weight matrices in deep neural networks, which allows us to explicitly track and control their singular values.
This parameterization does not restrict the network’s expressive power, while simultaneously allowing fast forward as well as backward propagation.
The method is easy to implement and has the same time and space complexity as compared to original methods like RNN and MLP.
The ability to control singular values helps in avoiding the gradient vanishing and exploding problems, and as we have empirically shown, gives good performance.
Although we only showed examples in the RNN and MLP framework, our method is applicable to many more deep networks, such as Convolutional
Networks etc.
However, further experimentation is required to fully understand the inﬂuence of using different number of reﬂectors in our SVD parameterization.
Also, the underlying structures of the image of M k 1 ,k 2 when k 1 , k 2 (cid:54) = 1 is a subject worth investigating.
A PPENDIX A P ROOFS A. P ROOF OF P ROPOSITION
Proposition 1.
(Householder QR factorization)
Let B ∈ R n × n .
Then there exists an upper triangular matrix R with positive diagonal elements, and vectors { u i } ni =1 with u i ∈
R i , such that
B = H nn ( u n ) ...
H n 1 ( u 1 ) R .
(Note that we allow u i = 0 , in which case, H ni ( u i ) =
I n as in (1) )
Proof of Proposition 1.
For n = 1 , note that H 11 ( u 1 ) = ± 1 .
By setting u 1 = 0 if B 1 , 1 > 0 and u 1 (cid:54) =
otherwise, we have the factorization desired.
Assume that the result holds for n =
k , then for n = k + 1 set u k +
=
B 1 − (cid:107) B 1 (cid:107) e 1 .
Here B 1 is the ﬁrst column of B and e 1 = (1 , 0 , ..., 0) (cid:62) .
Thus we have
[Equation]
, where ˆ B ∈ R k × k .
Note that H k +1 k +1 ( u k +1 )
=
I k
+1 when u k +
=
and the above still holds.
By assumption we have ˆ B
=
H kk ( u k ) ...
H k 1 ( u 1 ) ˆ R .
Notice that
H k
+1 i ( u i ) =
[Equation]
i ) (cid:17) , so we have that
[Equation]
is an upper triangular matrix with positive diagonal elements.
Thus the result holds for any n by the theory of mathematical induction.
A. P ROOF OF T HEOREM
Proof.
Observe that the image of M 1 is a subset of O ( n ) , and we now show that the converse is also true.
Given A ∈ O ( n ) , by Proposition 1, there exists an upper triangular matrix R with positive diagonal elements, and an orthogonal matrix Q expressed as Q = H nn ( u n ) ...
H n 1 ( u 1 ) for some set of Householder vectors { u i } ni =1 , such that A = QR .
Since A is orthogonal, we have
A
[Equation]
=
I n , thus:
[Equation]
Thus R is orthogonal and upper triangular matrix with positive diagonal elements.
So R = I n and
A = Q = H nn ( u n ) ...
H n 1 ( u 1 ) .
A. P ROOF OF T HEOREM
Proof.
It is easy to see that the image of M 1 , 1 is a subset of R n × n .
For any W ∈ R n × n , we have its
SVD, W = U Σ V (cid:62) , where Σ = diag ( σ ) .
By Theorem 1, for any orthogonal matrix U, V ∈ R n × n , there exists { u i } ni =1 { v i } ni =1 such that U
=
M 1 ( u 1 , ..., u n ) and V = M 1 ( v 1 , ..., v n ) , then we have:
[Equation]
A.
P ROOF OF
T
HEOREM
Proof.
Let A ∈ R n × n be an orthogonal matrix.
By Theorem 1, there exist { a i } ni =1 , such that
A =
M 1 ( a 1 , ..., a n ) .
Since A (cid:62) is also orthogonal, for the same reason, there exist { b i } ni =1 , such that A (cid:62) =
M 1 ( b 1 , ..., b n ) .
Thus we have:
[Equation]
Observe that one of k 2 ≥ k 1 − 1 and k 1 ≥ k 2 − 1 must be true.
If k 2 ≥ k 1 − 1 , set
[Equation] [Equation]
and then we have:
[Equation]
Else, assign:
[Equation]
and then we have:
[Equation]
A. P ROOF OF T HEOREM
Proof.
It is easy to see that the image of M m,n ∗ , ∗ is a subset of R m × n .
For any W ∈ R m × n , we have its SVD, W = U Σ V (cid:62) , where Σ is an m × n diagonal matrix.
By Theorem 1, for any orthogonal matrix U ∈ R m × m , V ∈ R n × n , there exists { u i } mi =1 { v i } ni =1 such that U
= H mm ( u m ) ...
H m 1 ( u 1 ) and V = H nn ( v n ) ...
H n 1 ( v 1 ) .
By Lemma 1, if m < n we have:
[Equation]
Similarly, for n < m , we have:
[Equation]
H nn ( v n ) .
A. P ROOF OF C LAIM
Proof.
R [ W ] = E X∼D
[ f ( W ; X , y )]
=
[Equation]
A
[Equation]
= E (cid:2)
[Equation]
(cid:0) (
[Equation]
−
[Equation] [Equation]
[Equation] [Equation]
) (cid:62) ( Y W − A ) (cid:62) (cid:1)(cid:3) + E [ (cid:107) η (cid:107) 2 ] − 2 E
[ (cid:104)
( Y W − A ) vec ( X ) , η (cid:105) ]
= tr (cid:0) ( Y W − A ) E (cid:2) vec ( X ) vec ( X ) (cid:62) (cid:3) ( Y W − A ) (cid:62) (cid:1) + C = (cid:107) ( Y W − A )Σ 1 / 2 (cid:107) 2 F + C
For the derivative,
R [ W + ∆ W ] = (cid:107) ( Y (cid:0) W + ∆ W | ( W + ∆ W ) 2 | · · · | ( W + ∆ W ) t (cid:1) − A )Σ 1 / 2 (cid:107) 2 F + C
=
R
[ W ] + (cid:68)(cid:0) ∆ W | 2 W ∆ W | · · · | tW t − 1 ∆ W (cid:1) Σ 1 / 2 , ( Y W − A )Σ 1 / 2 (cid:69) + o ( (cid:107) ∆ W (cid:107) 2 F ) =
R [ W ] + (cid:68) ∆ W, ( Y W − A )Σ (cid:0) I d | 2 W | 3 W 2 | · · · | tW t − 1 (cid:1) (cid:62) (cid:69)
Therefore ∇ R [ W ] =
( Y W − A )Σ (cid:0) I d | 2 W | 3 W 2 | · ·
[Equation]
Remark: here when W and ∆ W are not commutative, each W i ∆ W should instead be written as
[Equation]
ij
[Equation]
∆
[Equation]
i
[Equation]
Since the change of order doesn’t impact the analysis, we informally simplify the expressions here.
A
A LGORITHMS
Algorithm
Input : h ( t − 1) , ∂L∂C ( t ) , U = ( u n | ... | u n − m 1 +1 ) , Σ , V = ( v n | ... | v n − m 2 +1 )
Output : C ( t ) =
Wh ( t − 1) , ∂L∂U , ∂L∂V , ∂L∂ ˆ σ , ∂L ∂h
( t − 1) //
Begin forward propagation h ( v ) n +1 ← h ( t − 1) for k = n, n − 1 , ..., n − m
+ 1 do h ( v )
k ← Hprod ( h ( v ) k
+1 , v k ) //
Compute ˆ V (cid:62) h end for h ( u ) k 1 − 1 ← Σ h ( v ) k// Compute Σ
ˆ V (cid:62) h for k = n − m
+ 1 , ..., n do h ( u ) k ← Hprod ( h ( u ) k − 1 , u k ) //
Compute ˆ U Σ
ˆ V (cid:62)
h end for
C ( t ) ← h ( u ) n //Begin backward propagation g ← ∂L∂C
( t ) for k = n, n − 1 , ..., n − m 1 + 1 do g, G ( u ) ∗ ,n − k +1 ← Hgrad ( h ( u ) k , u k , g ) //
Compute ∂L∂u k end for ¯Σ ← diag ( g ◦ h ( v ) k 2 ) , g ← Σ g // Compute ∂L∂ Σ g (ˆ σ ) ← ∂diag (Σ) ∂
ˆ σ ◦ diag (¯Σ) //
Compute ∂L∂ ˆ σ for k = n − m 2 + 1 , ..., n do g, G ( v ) ∗ ,n − k +1 ← Hgrad
( h ( u ) k
+1 , v k , g ) //
Compute ∂L∂v k end for ∂L∂U ← G ( u ) , ∂L∂V
← G ( v ) , ∂L∂ ˆ σ ← g (ˆ σ ) , ∂L ∂h ( t − 1) ← g
Algorithm
ˆ h
=
Hprod ( h, u k )
Input : h, u k
Output : ˆ h =
H k
( u k ) h //
Compute ˆ h
= ( I − 2 u k u (cid:62) k u (cid:62)
k
u k ) h
α ← 2 (cid:107) u k (cid:107) 2 u (cid:62) k
h
ˆ h ← h − αu k
Algorithm 3 ¯ h, ¯ u k
= Hgrad ( h, u k , g )
Input : h, u k , g = ∂L∂C
where
C =
H k
( u k ) h
Output : ¯ h = ∂L∂h , ¯ u k = ∂L∂u
k
α = 2 (cid:107) u k (cid:107) 2 u (cid:62) k
h β = 2 (cid:107) u k (cid:107) 2 u (cid:62)
k
g ¯ h ← g − βu k ¯ u k ← − αg − βh + αβu k
A PPENDIX C M ORE E XPERIMENTAL D ETAILS C. D ETAILS ON THE T IME S ERIES C LASSIFICATION T ASK
For the time series classiﬁcation task, we use the training and testing sets directly from the UCR time series archive http://www.cs.ucr.edu/˜eamonn/time_series_data/ , and randomly choose 20% of the training set as validation data.
We provide the statistical descriptions of the datasets and experimental results in Table 3. [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
C. E XPERIMENTAL R ESULTS ON THE A DDING AND C OPYING TASKS
We tested RNN models on the Adding and Copying tasks with the same settings as Arjovsky et al.
C.2. A DDING TASK
The Adding task requires the network to remember two marked numbers in a long sequence and add them.
Each input data includes two sequences: top sequence whose values are sampled uniformly from [0 , 1] and bottom sequence which is a binary sequence with only two 1’s.
The network is asked to output the sum of the two values.
From the empirical results in Figure 4, we can see that when the network is not deep (number of layers L =30 in (a)(d)), every model outperforms the baseline of 0.167 (always output 1 regardless of the input).
Also, the ﬁrst layer gradients do not vanish for all models.
However, on longer sequences ( L =100 in (b)(e)), IRNN failed and LSTM converges much slower than svdRNN and oRNN.
If we further increase the sequence length ( L =300 in (c)(f)), only svdRNN and oRNN are able to beat the baseline within reasonable number of iterations.
We can also observe that the ﬁrst layer gradient of oRNN/svdRNN does not vanish regardless of the depth, while IRNN/LSTM’s gradient vanish as L becomes lager.
[Figure] [Figure]
C OPYING TASK
Let A = { a i } 9 i =
be the alphabet.
The input data sequence x ∈ A T +20 where T is the time lag.
x 1:10 are sampled uniformly from i { a i } 7 i =
and x T +10 is set to a 9 .
Rest of x i is set to a 8 .
The network is asked to output x 1:10 after seeing a 9 .
That is to copy x 1:10 from the beginning to the end with time lag T .
A baseline strategy is to predict a 8 for T +10 entrees and randomly sample from { a i } 7 i =1 for the last 10 digits.
From the empirical results in Figure 5, svdRNN consistently outperforms all other models.
IRNN and LSTM models are not able to beat the baseline with large time lag. In fact, the loss of RNN/LSTM is very close to the baseline (memoryless strategy) indicates that they do not memorize any useful information throughout the time lag.