U NDERSTANDING G ENERALIZATION IN R ECURRENT N EURAL N ETWORKS
A BSTRACT
In this work, we develop the theory for analyzing the generalization performance of recurrent neural networks.
We ﬁrst present a new generalization bound for recurrent neural networks based on matrix 1-norm and Fisher-Rao norm.
The deﬁnition of Fisher-Rao norm relies on a structural lemma about the gradient of
RNNs.
This new generalization bound assumes that the covariance matrix of the input data is positive deﬁnite, which might limit its use in practice.
To address this issue, we propose to add random noise to the input data and prove a generalization bound for training with random noise, which is an extension of the former one.
Compared with existing results, our generalization bounds have no explicit dependency on the size of networks.
We also discover that Fisher-Rao norm for RNNs can be interpreted as a measure of gradient, and incorporating this gradient measure not only can tighten the bound, but allows us to build a relationship between generalization and trainability.
Based on the bound, we theoretically analyze the effect of covariance of features on generalization of RNNs and discuss how weight decay and gradient clipping in the training can help improve generalization.
I NTRODUCTION
The Recurrent Neural network (RNN) is a neural sequence model that has achieved the state-ofthe-art performance on numerous tasks, including natural language processing (Yang et al., 2018; Mikolov & Zweig, 2012), speech recognition (Chiu et al., 2018; Graves, 2013) and machine translation (Wu et al., 2016; Kalchbrenner & Blunsom, 2013).
Unlike feedforward neural networks, RNNs allow connections among hidden units associated with a time delay.
Through these connections, RNNs can maintain a ”memory” that summarizes the past sequence of inputs, enabling it to capture correlations between temporally distant events in the data.
RNNs are very powerful, and empirical studies have shown that they have a very good generalization property.
For example, Graves (2013) showed that deep LSTM RNNs achieved a test error of 17 .
7% on TIMT phoneme recognition benchmark after training with only 462 speech samples.
Despite of the popularity of RNNs in practice, their theory is still not well understood.
A number of recent works have sought to shed light on the effective representational properties of recurrent networks trained in practice.
For example, Oymak (2018) studied the state equation of recurrent neural networks and showed that SGD can efﬁciently learn the unknown dynamics from few observations under proper assumptions.
Miller & Hardt (2019) tried to explain why feed-forward neural networks are competitive with recurrent networks in practice.
They identiﬁed stability as a necessary condition and proved that stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent.
Despite of the impressive progress in understanding the training behavior of RNNs, there is no generalization guarantee in these works.
Understanding the generalization performance in machine learning has been a central problem for many years and revived in recent years with the advent of deep learning.
One classical approach to proving generalization bound is via notions of complexity.
For deep neural networks, numerous complexity measures have been proposed to capture the generalization behavior such as VC dimension (Harvey et al., 2017) and norm-based capacity including spectral norm (Bartlett et al., 2017; Neyshabur et al., 2019), Frobenius norm (Neyshabur et al., 2015b;a; 2018) and l p -path norm (Neyshabur et al., 2015b; Bartlett & Mendelson, 2002; Golowich et al., 2018).
These existing normbased complexity measures depend on the number of hidden units of the network explicitly and thus can not explain why neural networks generalize so well in practice, despite that they operate in an overparametrized setting (Zhang et al., 2017).
Neyshabur et al.
(2019) proved generalization bounds for two layer ReLU feedforward networks, which decreased with the increasing number of hidden unit in the network.
However their results only applied to two layer ReLU networks and some speciﬁc experiments.
More recently, a new generalization bound based on Fisher-Rao norm was proposed (Liang et al., 2017).
This notion of Fisher-Rao norm is motivated by information geometry and has good invariance properties.
But they proved the bound only for linear deep neural networks.
There are also some works about the generalization of recurrent neural networks (Zhang et al., 2018; Chen et al., 2019; Allen-Zhu & Li, 2019).
However these bounds also depend on the size of networks, which makes them vacuous for very large neural networks.
Our main contributions are summarized as follows
• We deﬁne the Fisher-Rao norm for RNNs based on its gradient structure and derive new Rademacher complexity bound and generalization bound for recurrent neural networks based on Fisher-Rao norm and matrix 1-norm.
In contrast to existing results such as spectral norm-based bounds, our bound has no explicit dependence on the size of networks.
• We prove a generalization bound for RNNs when training with random noises.
Our bound applies to general types of noises and can potentially explain the effect of noise training on generalization of recurrent neural networks as demonstrated by our empirical results.
• We propose a new technique to decompose RNNs with ReLU activation into a sum of linear network and difference terms.
As a result, each term in the decomposition can be treated independently and easily when estimating the Rademacher complexity.
This decomposition technique can potentially be applied to other neural networks architectures such as convolutional neural networks, which might be of independent interest.
The remainder of this paper is structured as follows.
We deﬁne the problem and notations in Section 2.
The notion of Fisher-Rao norm for RNNs is introduced in Section 3.1.
We prove the generalization bound for RNNs in Section 3.2, and the generalization bound for training with random noise is derived in Section 3.3.
Section 3.4 gives a detailed analysis of our generalization bound.
Finally we conclude and discuss future directions.
P RELIMINARIES
We focus on the vanilla RNNs with ReLU activation.
Let U ∈ R m × d , V ∈ R k × m and W ∈ R m × m be the weight matrices.
Given the input sequence x = ( x 1 , x 2 , · · · , x L ) ∈ R Ld where each x i ∈ R d and L is the input sequence length, the vanilla RNNs can be described as follows.
[Equation]
where g t and h t ∈ R m represents the input and output of hidden layer at step t , ρ ( · ) is the ReLU function and
y t ∈ R k denotes the output value at step t .
For simplicity, in this paper, we only consider the ﬁnal output y L .
We assume that data ( x, y ) is drawn i.i.d.
from some unknown distribution D over R Ld × Y where Y represents the label space { 1 , 2 , · · · , k } .
The RNNs above deﬁne a mapping y L ( x ) from R Ld → R k , where k is the number of classes.
We convert y L ( x ) to a classiﬁer by selecting the output coordinate with the largest magnitude, meaning
[Equation]
i [Equation] where [ · ] i represents the i -th element of a vector.
This naturally leads to the deﬁnition of margin
M y L ( x, y ) of the output y L at a labeled example ( x, y ) :
[Equation]
] y (cid:48) .
Thus, y L misclassiﬁes ( x, y ) if and only if M y L ( x, y ) ≤ 0 .
The quality of the prediction made by y L is measured by the expected risk deﬁned as
[Equation]
Since the underlying distribution D is unknown to us, we instead consider the empirical error on sample data given by
[Equation]
The generalization error is then the difference between expected risk and empirical risk, deﬁned as
[Equation]
Our goal in this paper is to study the generalization error for RNNs theoretically.
To establish the generalization bound, a little bit of notations are necessary.
For a vector, we denote the l p norm by || v || p
=
( (cid:80) | v i | p ) 1 /p
and the l ∞ norm by || v || ∞ = max | v i | .
For a matrix, we denote the matrix p -norm as || A || p = max
|| x || p =1 ||
Ax || p , the matrix 1-norm by ||
A || 1 = max j { (cid:80) i | a ij |} and the Frobenius norm by || A || 2 F = trace ( AA T ) .
The smallest eigenvalue of a matrix A is given by λ min ( A ) .
The activation function ρ and its derivative ρ (cid:48) are entrywise, i.e., ρ ( A ) =
( ρ ( a ij )) ij and ρ (cid:48) ( v ) =
( ρ (cid:48) ( v i ))
i .
We denote c =
( L + 1 , L, · · · , 2) T , η ( θ ) =
[ V diag ( ρ (cid:48) ( g L )) ...
Wdiag ( ρ (cid:48) ( g 1 ))
Ux 1 , V diag ( ρ (cid:48) ( g L )) ...
Wdiag ( ρ (cid:48) ( g 2 ))
Ux 2 , · · · , V diag ( ρ (cid:48) ( g L ))
Ux L ] ∈ R k × L and τ ( θ ) =
( V W L −
Ux 1 , V W L −
Ux 2 , · · · , V Ux L ) where θ =
( U, W, V ) and diag converts a vector into a diagonal matrix.
M AIN R ESULT
In this section, we prove a generalization bound for RNNs with ReLU activation.
Our new bound is based on Fisher-Rao norm and matrix 1-norm.
We ﬁrst deﬁne the Fisher-Rao norm for RNNs.
3. F ISHER -R AO N ORM FOR RNN S
We adapt the notion of Fisher Rao norm to recurrent neural networks.
To begin with, we establish the following structural result for RNNs.
Lemma 1.
Given an input x = ( x 1 , x 2 , · · · , x L ) , consider the recurrent neural network in (1), we have the identity (cid:88)
[Equation]
The notion of Fisher-Rao norm is motivated by Fisher-Rao metric of information geometry and is deﬁned as follows.
Deﬁnition 1 ((Liang et al., 2017), Deﬁnition 2) .
The Fisher-Rao norm for a parameter θ is deﬁned as
[Equation]
where I ( θ ) =
E ( ∇ l ( y Lθ ( x ) , y ) ⊗ ∇ l ( y Lθ ( x ) , y )) and l ( ., . )
is the loss function.
The following lemma gives the explicit formula of Fisher-Rao norm in RNNs.
We can see that the notion of Fisher-Rao norm relies mainly on the gradient structure of RNNs.
Lemma 2.
Assume that the loss function l ( ., . ) is smooth in the ﬁrst argument.
Then the following identity holds for the RNN in (1),
[Equation]
Remark 1.
We observe that each term V diag ( ρ (cid:48) ( g L )) ...
Wdiag ( ρ (cid:48) ( g i ))
Ux i in η ( θ ) is actually the gradient component in Backpropagation through time (BPTT).
Therefore, the Fisher-Rao norm can be regarded as a measure of gradient.
As will be shown later, we can build a relationship between generalization and trainability in RNNs via Fisher-Rao norm.
For the linear activation function and margin loss l ( y Lθ ( x ) , y )
=
Φ α ( M y L ( x, y )) where α > 0 is the margin parameter, one might upper bound the Fisher-Rao norm in Lemma 2 by
[Equation]
since (cid:10) τ ( θ ) c, ∂l ( y Lθ ( x ) , y )
∂y Lθ (cid:11) 2 ≤ 4 α 2 max i [( τ ( θ ) c ) i ] 2 by deﬁnition of M y L ( x, y ) and lipschitz property of Φ α ( · ) .
We deﬁne this upper bound as
[Equation]
and still call it ”Fisher-Rao norm” in the paper by slightly abusing the terminology as they are equivalent for k = 1 .
In the rest of the paper, we will use this Fisher-Rao norm || · || fs to derive generalization bound for RNNs.
3. G ENERALIZATION B OUND FOR RNN S
We use matrix 1-norm and Fish-Rao norm together to derive a generalization bound for RNNs.
Since it is very challenging to bound the Radermacher complexity of ReLU networks directly in terms of the Fisher-Rao norm, we consider decomposing the ReLU network into the sum of a linear network and a difference term, i.e., y L = ψ ( θ ) x
+ ( y L − ψ ( θ ) x ) .
For the linear network part ψ ( θ ) x , the Rademacher complexity can be bounded directly by Fisher-Rao norm.
For the difference term ( y L − ψ ( θ ) x ) , we further decompose it into a sum of simpler terms and then upper bound the Rademacher complexity of these simpler terms by matrix 1-norm.
We ﬁrst give the results for the linear network part.
Lemma 3.
Deﬁne F r :=
{ x → [ ψ ( θ ) x ] y :
||
θ || fs ≤ r, y ∈ Y} where x ∈ R Ld and ψ ( θ ) :
= ( V W L − 1 U, V W L − 2 U, · · · , V U ) .
For any data x 1 , x 2 , · · · , x n drawn i.i.d from the distribution D , collect them as columns of a matrix X ∈ R Ld × n .
Then we have
[Equation]
assuming that E ( xx T ) is positive deﬁnite.
Remark 2.
If E ( x ) = 0 , E ( xx T ) is the covariance matrix of random variable x .
Remark 3.
We should mention that our assumption that E ( xx T ) is positive deﬁnite is not so restrictive and usually holds in practice.
For example, for the case that x is continuous random variable, we can prove that E ( xx T ) is positive deﬁnite as follows.
Suppose that x is a continuous random variable in the n -dimensional subspace X ⊂ R n .
If there exists u ∈ R n such that u T E ( xx T ) u = 0 , then for any x ∈ X we have u T x = 0 , i.e., u ⊥ X .
Since X is n -dimensional, the only u that satisﬁes is that u = 0 .
Therefore, by deﬁnition, E ( xx T ) is positive deﬁnite.
As we will show in Section 3.3, this assumption can be removed, and a more general generalization bound will be presented.
Now we bound the Rademacher complexity of the difference term y L − ψ ( θ ) x .
With a slight abuse of notations, given input data x 1 , x 2 , · · · , x n ∈ R Ld , the corresponding g 1 , g 2 , · · · , g n ∈ R Lm and h 1 , h 2 , · · · , h n ∈ R Lm are calculated by (1) .
We collect all input data as a matrix denoted by
X , all input data at time t as a matrix denoted by X t , all input of the hidden layer at time t as a matrix denoted by G t and all output of hidden layer at time t denoted by H t , where X ∈ R Ld × n ,
X t ∈ R d × n , G t ∈ R m × n , H t ∈ R m × n and t = 1 , · · · , L .
The difference term can be decomposed by the following lemma.
Lemma 4.
Deﬁne H (cid:48)(cid:48) t :=
H t − G t .
Then the following equality holds
[Equation]
To bound the Rademacher complexity of each term in the above decomposition, we need a technical lemma given as follows.
Lemma 5.
For any p ≥ 1 , | [Equation] H
[Equation]
As we will see, the operator norm in Lemma 5 will be instantiated for the case of p = 1 .
The use of || · || 1 helps avoid the appearance of the dimension m when upper bounding the Rademacher complexity.
Also it guarantees that Rademacher complexity has a convergence rate O (1 /n ) .
The upper bound for the Rademacher complexity of these individual term is given by the following lemma.
Lemma 6.
Let Ω :=
{ θ = ( U, W, V ) :
|| V T || 1 ≤ β V , || W T || 1 ≤ β W ,
||
U T || 1 ≤ β U } .
Then for any i = 1 , · · · , L , we have
[Equation]
where σ = ( σ 1 , σ 2 , · · · , σ n ) T is Rademacher random variable and [ · ] y, represents the y-th row of the matrix.
We are now ready to put the ingredients together to prove our ﬁrst theorem.
Theorem 1 (Rademacher complexity of RNNs) .
Let Ω :=
{ θ = ( U, W, V ) :
|| V T || 1 ≤ β V , || W T || 1 ≤ β W ,
||
U T || 1 ≤ β U ,
||
θ
|| fs ≤ r } .
Then, the empirical Rademacher complexity of
RNNs with ReLU can be bounded as follows
[Equation]
where Λ :=
[Equation]
To establish the generalization bound for RNNs, we need the following classical results for multiclass margin bounds.
Lemma 7 ((Kuznetsov et al., 2015), Theorem 2) .
Let H ⊆
R X×Y be a hypothesis set with Y = { 1 , 2 , · · · , k } .
Fix α > 0 .
Then, for any δ > 0 , with probability at least 1 − δ , the following multi-class classiﬁcation generalization bound holds for all h ∈ H :
[Equation]
where Π 1 ( H ) = { x → h ( x, y ) : y ∈ Y , h ∈ H } .
The generalization bound for RNNs follows from combining Theorem 1 and Lemma 7.
Theorem 2.
Fix margin parameter α , then for any δ > 0 , with probability at least 1 − δ , the following holds for every RNN whose weight matrices θ
= ( U, W, V ) satisfy || V T
|| 1 ≤ β V , || W T
|| 1 ≤ β W ,
||
U T || 1 ≤ β U
and ||
θ || fs ≤ r :
[Equation]
Comparison with existing results.
We compare our result with the existing generalization bounds (Zhang et al., 2018; Chen et al., 2019).
In comparison with the bound in Zhang et al.
(2018), which is of the order ˜ O (max { d, m, k } L
2 ||
U
|| 2 ||
V || 2 max { 1 , ||
W || L 2 } √ nα ) , there is no explicit appearance of the network size parameters d and m in our bound.
As we have mentioned before, the reason that we can avoid these dimensional factors is that we use matrix 1-norm instead of spectral norm to upper bound the Rademacher complexity of the network.
Moreover, there is always a L 2 factor in their bound, whereas the L 2 term only occurs in our bound when || W T || 1 = 1 .
For the case that || W T || > 1 , our bound has only a linear dependence on L , and for the case that || W T
|| 1 < 1 , by simple calculation, we can show that Λ ≤ 1 (1 − β W ) 2 and the dependence on L would vanish.
Both of our bounds have an exponential term || W || L , which would make the bounds become vacuous for || W || > 1 .
It should also be pointed out that our bound scales linearly with the number of classes since we handle multiclass on each coordinate of a k -tuple of functions and pay a factor of k .
Chen et al.
(2019) also derived generalization bound for RNNs in terms of spectral norm and the total number of parameters of the network by using covering number analysis.
Since their work assumed that the activation function in the hidden layers was bounded rather than the ReLU activation function considered in our paper, their bound is not directly comparable to ours, and we do not make a comparison here due to the page limit.
We should emphasis that our proof technique is totally different from the PAC-Bayes approach (Zhang et al., 2018) and covering number analysis (Chen et al., 2019).
In particular, we work on the Rademacher complexity of RNNs directly with no invocation of complicated tools such as covering number, which makes our analysis conceptually much simpler.
There is also an additional bonus of our proof technique.
In the next section, we will use this proof technique to derive a generalization bound for RNNs when training with random noise.
3. G ENERALIZATION B OUND FOR T RAINING WITH R ANDOM N OISE
The generalization bound in Theorem 2 requires the input covariance matrix E ( xx T ) to be positive deﬁnite and would become very poor when the smallest eigenvalue is close to 0, which greatly limits the power of our bound.
To address this issue, we consider adding random noise to the input data.
We notice that after adding random noise with mean 0 and variance σ 2 (cid:15) , the term E ( xx T ) in the bound becomes E (( x + (cid:15) )
( x + (cid:15) ) T ) and the smallest eigenvalue of E (( x + (cid:15) )
( x + (cid:15) ) T ) is ( λ min ( E ( xx T ))
+ σ 2 (cid:15) ) , which is greater than σ 2 (cid:15) .
Therefore, our bound is still applicable even when the covariance matrix of original input data is rank-deﬁcient.
Involving noise variables has been widely used in recurrent neural networks as a regularization technique (Bayer et al., 2013; Zaremba et al., 2014; Dieng et al., 2018; Gal & Ghahramani, 2016).
For example, Bayer et al.
(2013) claimed that conventional dropout did not work well with RNNs because the recurrence ampliﬁed noise, which in turn hurt learning.
To ﬁx this problem, Zaremba et al.
(2014) proposed to inject noise only to the input and output of RNNs.
Although their method greatly reduced overﬁtting on a variety of tasks, the generalization guarantee was not provided.
In this section, we present a generalization bound for RNNs with noise training.
For simplicity, we assume that the noise is drawn i.i.d.
from a Gaussian distribution with zero mean and variance σ 2 (cid:15) .
Let (cid:15) i denotes the d -dimensional gaussian noise generated at step i and (cid:15) =
( (cid:15) 1 , (cid:15) 2 , · · · , (cid:15) L
) ∈
R Ld .
We collect all noise data as a matrix denoted by X (cid:15) .
To prove the generalization bound, we need to use the Lipschitz property of RNNs given by the following lemma.
Lemma 8.
For every RNN in (1) with weight matrices θ =
( U, W, V ) , y L is Lipschitz with respect to || · || ∞ , i.e.,
[Equation]
for any x = ( x 1 , x 2 , · · · , x L ) , x (cid:48) =
( x (cid:48) 1 , x (cid:48) 2 , · · · , x (cid:48)
L ) ∈ R Ld .
The generalization bound for training with random noise is described as follows.
Theorem 3.
Fix margin parameter α , then for any δ > 0 , with probability at least 1 − δ over a sample (( x 1 , (cid:15) 1 , y 1 ) , ( x 2 , (cid:15) 2 , y 2 ) , · · · , ( x n , (cid:15) n , y n )) , the following holds for every RNN whose weight matrices θ
= ( U, W, V ) satisfy || V T
|| 1 ≤ β V , || W T
|| 1 ≤ β W ,
||
U T || 1 ≤ β U
and ||
θ || fs ≤ r :
[Equation]
Remark 4.
The above bound can be easily extended to other kinds of noises by replacing σ (cid:15) (cid:112) 2 log(2 d ) by E (cid:15)
|| (cid:15)
i || ∞ .
Remark 5.
The bound in Theorem 3 is an extension of that in Theorem 2 and can be applied even when the smallest eigenvalue of E ( xx T ) is very close to 0 .
For example, when λ min ( E ( xx T )) = 1 × 10 − 6 , applying Theorem 2 directly might lead to a vacuous bound.
But if using Theorem 3 by choosing a small noise with mean 0 and variance 0.01, we might obtain a better bound since the term (cid:113) 1 λ min ( E (( xx T )))
+ σ 2 (cid:15) ≤ 10 .
Notice that adding noise can not always guarantee an improved generalization especially when λ min ( E ( xx T )) is not so small as it incurs an additional linear term 2 α (cid:80)
i β
V β U β L − i W σ (cid:15) (cid:112) 2 log(2 d ) and might also increase other parameters in the bound such as || X + X (cid:15)
|| F .
Therefore we suggest adding noise only when the smallest eigenvalue of E ( xx T ) is very small.
For this case, a small noise such as σ (cid:15) = 0 .
1 not only can greatly improve the term (cid:114) λ min ( E ( xx T )) but also ensure that the extra cost σ (cid:15) (cid:112) 2 log(2 d ) and || X + X (cid:15)
||
F /n be small enough since || X + X (cid:15)
|| F /n ≤
|| X
|| F /n
+
|| X (cid:15)
||
F /n and || X (cid:15)
||
F /n would be small when n is large.
Remark 6.
If we remove the constraint condition ||
θ || fs ≤ r , which means that we do not have any knowledge about the gradients, the generalization bound in Theoremand Theorem 3 still holds by substituting r with β V β U B ( [Equation] ) for β W < 1 .
But with this extra gradient measure, the bound can become much tighter, especially when λ min ( E ( xx T ) is small.
Please refer to the detailed analysis in the next section.
± standard error averaged on 5 runs).
Experiments.
We now study the effect of random noise on generalization of RNNs empirically.
For simplicity, we consider the IMDB dataset, a collection of 50K movie reviews for binary sentiment classiﬁcation.
We use GloVe word embedding to map each word to a 50dimensional vector.
We train vanilla RNNs with ReLU activation function for sequence length L = 100 .
The corresponding smallest eigenvalue of E ( xx T ) is approximated by using the total training data, which is 4 × 10 − 4 .
We add Gaussian noise to the input data in the training process with σ (cid:15) = 0 .
1 , 0 . 2 , 0 . 3 and 0 .
The generalization error which is the gap between test error without noise and training error with noise for L = 100 and different values of σ (cid:15) is shown in Figure 1 (results for other values of L in Appendix D).
We observe that as we start injecting noise, the generalization error becomes better.
But when the deviation of noise keeps growing, the generalization error shows an increasing tendency.
This behavior is consistent with the prediction made by our bound.
3. A NALYSIS OF G ENERALIZATION B OUND
Our theoretical results have a number of implications for the generalization performance in RNNs, and some of them have been observed in empirical studies.
We summarize these implications as follows.
3.4. G ENERALIZATION AND S MALLEST E IGENVALUE OF E ( xx T )
According to our results, the generalization performance in RNNs is inﬂuenced by the smallest eigenvalue of E ( xx T ) .
Since the smaller eigenvalues may contribute to high frequency components of the input signal, our bound suggests that high frequency information is potentially more difﬁcult to generalize, which is consistent with intuition.
There are many factors that impact on the smallest eigenvalue and therefore the generalization performance in RNNs.
In particular, we study the effect of the correlation between features on the generalization in RNNs.
The exact answer for this problem may be complicated.
Here we only make an initial attempt and claim that weaker correlation would help improve the generalization, and a non-rigorous proof is given as follows.
Denote the covariance matrix E
( xx T ) by Ξ where each element ξ ij in Ξ represents the covariance between feature i and j .
Suppose that || Ξ − I || 1 ≤ ζ with ζ < 1 .
By deﬁnition of || · || 1 matrix norm, we immediately get | ξ ii − 1 | + (cid:80) j (cid:54) =
i | ξ ij | ≤ ζ for any i .
Then by simple derivation, we obtain ξ ii − (cid:80) j (cid:54) =
i | ξ ij | ≥ 1 − ζ for any i .
Applying Gershgorin circle theorem, we have that the smallest eigenvalue must be greater or equal than 1 − ζ .
Since the element ξ ij with i (cid:54) =
j represents the covariance between feature i and j , a weaker correlation between feature i and j means a smaller value of | ξ ij |
and we need a smaller ξ to upper bound
|| Ξ −
I
|| 1 , which gives us a bigger lower bound on the smallest eigenvalue.
Therefore the generalization bound becomes better.
3.4. G ENERALIZATION AND T RAINABILITY
The generalization of RNNs also depends on parameters β U , β V , β W and r , where β U , β V and β W control the weight matrices and r represents the gradient measure.
It has a natural relationship with the training process.
The normal procedure in training RNNs is to use weight decay for regularization and gradient clipping to avoid the exploding gradients problems (Bengio et al., 1994; Pascanu et al., 2013).
From the perspective of generalization, these strategies can decrease the value of these parameters β U , β V , β W and r and thus improves the generalization.
For example, if β W ≤ 1 , we have Λ ≤ 1 (1 − β W ) 2 , and the second term 1 n β V β U || X T || 1 Λ in the generalization bound would be small when β W is not so close to 1.
Similarly, if λ min ( E ( xx T )) is very small, by setting the gradient clipping value in the training procedure, we can achieve a smaller value of r and thus good generalization.
Therefore our bound partially explains why training RNNs in this way can achieve good performance in practice.
3.4. G ENERALIZATION AND G RADIENT M EASURE
We are interested in how the gradient measure contributes to generalization.
Suppose now that we only have the weights, i.e., the parameters β U , β W and β V and the gradient measure parameterized by r is unknown to us.
To apply our bound, a natural idea is to infer the gradient measure parameter r based on the known weight parameters.
An upper bound for r in terms of β U , β W and β V is given as follows.
Under the same conditions as Theorem 2, if we further assume that the data x be given with || x T || 1 ≤ B , by the deﬁnition of || · || fs in (2), for any y ∈ Y , we have
[Equation]
, for β W < 1 , and (( τ ( θ ) c ) y (cid:1) 2 ≤ ( β V β U B 3 L + L 2) 2 for β W = 1 .
The above inequality holds for any x and y .
So we can get ||
θ
|| fs = E [Equation]
[Equation]
U B ( 1 (1 − β W ) 2 + 1 1 − β W ) for β W < 1 .
By replacing r with β V β U B ( 1 (1 − β W ) 2 + 1 1 − β W )
, the generalization bound (3) also holds.
But notice that this bound is obtained without any knowledge about the gradients.
If we happen to know that the parameter r is much smaller than β V β U B ( 1 (1 − β W )
+ 1 1 − β W ) , for example, by setting gradient clipping value to be small in training process, this extra gradient measure can provide us with a better generalization bound, especially when the smallest eigenvalue of E ( xx T ) is small.
Therefore the introduction of Fisher-Rao norm can help eliminate the negative effect of λ min ( E ( xx T )) and thus improve the generalization bound.
C ONCLUSION
In this paper, we propose a new generalization bound for RNNs in terms of matrix 1-norm and
Fisher-Rao norm, which has no explicit dependence on the size of networks.
Based on the bound, we analyze the inﬂuence of covariance of features on generalization of RNNs and discuss how weight decay and gradient clipping in the training can help improve generalization.
While our bound is useful for analyzing generalization performance of RNNs, it would become vacuous because of an exponential term when || W T || 1 > 1 .
It is of interest to get a tighter bound which can avoid this exponential dependence.
Moreover, our bound only applies to vanilla RNNs with ReLU activation, and extending the results to tangent and sigmoid activation functions or other variants of RNNs like
LSTM and MGU might be an interesting topic for future research.
A CKNOWLEDGMENTS
This work was supported by Australian Research Council Project FL-170100117.
We would like to thank Gemeng Zhang from Tulane University for helpful discussions and all the reviewers for their constructive comments
A P ROOFS IN S ECTION 3. A. P ROOF OF L EMMA
Proof.
To begin with, by equation (1)
[Equation]
we have y L = V h L .
Then the derivative of y L with respect to v ab can be calculated as
[Equation]
i
[Equation]
e., a k -dimensional vector with a -th element [ h L ] b and all other elements zero.
Multiplying both sides by v ab and summing them up, we get (cid:88)
[Equation]
The derivative of y L with respect to W and U can be derived by using chain rule in the similar way as follows.
[Equation]
and
[Equation]
where we use the property of ReLU activation function that ρ ( z ) =
ρ (cid:48) ( z ) z .
Summing up these terms immediately gives us the following equality
[Equation]
For ease of exposition, deﬁne f L :=
[Equation]
∂u pq u pq .
Then the above equality can be rewritten as
[Equation]
By induction
[Equation]
we have
[Equation]
Multiplying both sides by V and using some basic calculation
[Equation]
we can show that V f L =
Ly L − η ( θ )(0 , 1 , · · · , L − 1) T .
Therefore,
[Equation]
Substituting y L = η ( θ )(1 , 1 , · · · , 1) T into the above equality leads to the desired result
[Equation]
A. P ROOF OF L EMMA
Proof.
Using the deﬁnition of Fisher-Rao norm,
[Equation]
By Lemma 1, we have ∇ y Lθ ( x ) T θ = η ( θ ) c .
Substituting it into the above equality gives us
[Equation]
B P ROOFS IN S ECTION 3. B. P ROOF OF L EMMA
The proof of Lemma 3 relies on the following result in Saniuk & Rhodes (1987).
Proposition 1.
Let X, Y ∈ R n × n with Y symmetric and nonnegative deﬁnite.
Then,
[Equation]
Now we are ready to prove Lemma 3.
Proof.
Denote A := E ((( L + 1) x T 1 , Lx T 2 , · · · , 2 x TL ) T (( L + 1) x T 1 , Lx T 2 , · · · , 2 x TL )) .
By the deﬁnition of ||
θ || fs , for any y ∈ Y , we have
[Equation]
[ ψ ( θ )]
y,T = E ([ ψ ( θ )]
y, (( L + 1) x T 1 , Lx T 2 , · · · , 2 x TL ) T (( L + 1) x T 1 , Lx T 2 , · · · , 2 x TL )
[ ψ ( θ )]
y,T ) =
E ([( τ ( θ ) c ) y ] 2 ) ≤ || θ
||
2 fs , where [ ψ ( θ )]
y, represents the y- th row of ψ ( θ ) [Equation]
On the other hand, from the deﬁnition of
Rademacher complexities,
[Equation]
where the ﬁrst inequality uses Cauchy-Schwarz inequality and C :
= diag (( L +1) I d , LI d , · · · , 2 I d ) .
Since C − 1 and E ( xx T ) are positive deﬁnite, we have
[Equation]
where the ﬁrst inequality is by Proposition 1 and the last inequality uses trace ( XX T )
=
||
X || 2 F .
Therefore,
[Equation]
B. P ROOF OF L EMMA
Proof.
Denote H (cid:48) t :=
UX t + WH (cid:48)
t − 1 and H (cid:48) 1 :=
UX 1 .
By the deﬁnition of H L , we have
H L
[Equation]
H (cid:48)
L =
[Equation]
UX
[Equation]
which by induction gives
[Equation]
So the difference term can be rewritten as
[Equation]
where the second equality uses ψ ( θ ) X
= V H
[Equation]
B. P ROOF OF L EMMA
Proof.
Using Riesz-Thorin Theorem, we have || H (cid:48)(cid:48)
t ||
p ≤ || H (cid:48)(cid:48)
t || 1 /p
|| H (cid:48)(cid:48) t
|| 1 − 1 /p ∞ .
And since H (cid:48)(cid:48) t = ρ ( G t ) − G t , by the deﬁnition of the induced L 1 and L ∞ matrix norm, we know
|| H (cid:48)(cid:48)
t || 1 ≤
|| G t
||and || H (cid:48)(cid:48) t || ∞
≤ || G t || ∞ .
Therefore,
[Equation]
where the last inequality uses some basic facts about matrix norm
that ||
G t || 1 ≤ m 1 − 1 /p
||
G t
|| p
and
|| G t || ∞ ≤ n 1 /p
||
G t || p .
B. P ROOF OF L EMMA
Proof.
For any y ∈ Y
[Equation]
by H ¨ older’s inequality
[Equation]
for any p
[Equation]
q ≥ 1 with 1 p + 1 q = 1 ,
[Equation]
In order to eliminate the dimension dependency on m and simultaneously enjoy a faster convergence rate with respect to n
[Equation]
we choose p = 1 .
Then the above inequality reduces to 1 n [ V
[Equation]
L [Equation] [Equation]
[Equation]
≤ ||
[ V ]
[Equation]
, T || 1 || W T
|| L − in −
1 ||
G i T
||
≤ β V β L − i W n − 1 ||
G i T
||[-
-]
[Equation] β V β L − i W n − 1 ( β U
|| X iT
||
1 + β W
|| H
i − 1 T || 1 ) .
For || H i − 1 T
|| 1 , we have
[Equation]
which by induction gives
[Equation]
Therefore,
E
[Equation]
sup
[Equation]
B. P ROOF OF T HEOREM
Proof.
Using the notations that we have introduced earlier
[Equation]
the empirical Rademacher complexity can be rewritten as
[Equation]
where the second equality uses Lemma 4 and the last inequality is due to the fact that Ω ⊆ Ω and Ω ⊆ { θ : ||
θ
|| fs ≤ r } .
For the last term
[Equation]
by Lemma
[Equation]
we have
[Equation]
The other terms can be handled by Lemma 6 in the following way.
[Equation]
for β W (cid:54) =
[Equation]
and (cid:80) Li
=1 E σ (cid:0) sup θ ∈ Ω [Equation] y
[Equation]
B
[Equation]
2for β W = 1 , where the second inequality uses the deﬁnition of matrix norm || · || 1 .
Collecting all terms
[Equation]
we establish
[Equation]
C P ROOFS IN S ECTION 3.
This section includes the full proofs of the generalization bound for training with random noise.
L IPSCHITZ PROPERTIES OF R E LU NONLINEARITIES AND M ARGIN OPERATOR
We ﬁrst establish the Lipschitz properties of the ReLU activation function and margin operator
M ( y L ( x ) , y ) :=
M y L ( x, y ) .
Lemma 9.
Let ρ : R n → R n be the coordinate-wise ReLU function, then it is 1 -Lipschitz according to || · || p for any p ≥ 1 .
Proof.
For any x, x (cid:48) ∈ R n ,
[Equation]
Lemma 10.
For every j and every p ≥ 1 , M ( · , j ) is 2 -Lipschitz wrt || · || p .
Proof.
Let y, y (cid:48) , j be given, and suppose that M ( y, j ) ≤ M ( y (cid:48) , j ) without loss of generality.
Select coordinate i (cid:54) =
j so that M ( y, j ) =
y j − y i .
Then
M
[Equation]
C. P ROOF OF L EMMA
Proof.
We prove this Lemma by induction.
Let x = ( x 1 , x 2 , · · · , x L ) and x (cid:48) =
( x (cid:48) 1 , x (cid:48) 2 , · · · , x (cid:48) L ) .
Denote g (cid:48) t :=
Ux (cid:48) t + Wh (cid:48)
t − 1 , h (cid:48) t :
= ρ ( g (cid:48) t ) and y (cid:48) t :
= V h (cid:48) t .
Then we have
[Equation]
| ∞ ≤
[Equation]
∞ , where the ﬁrst inequality uses Lemma 9 and the second inequality uses basic properties of || · || ∞ .
By induction, we get
[Equation]
Therefore,
[Equation]
C. P ROOF OF T HEOREM
We begin by establishing two auxiliary lemmas that we will need for the subsequent theorem.
Lemma 11.
For every RNNs in (1) with weight matrices θ =
( U, W, V ) , the following inequality holds for any x = ( x 1 , x 2 , · · · , x L ) and y .
[Equation]
Proof.
For any ﬁxed x and y ,
[Equation]
where the ﬁrst inequality uses Jensen’s inequality, the second inequality follows from the 1 αLipschitz property of Φ α ( · ) and Lemma 10 and the last inequality is by Lemma 8.
Lemma 12.
Let { (cid:15) i } di =1 be an i.i.d sequence of N (0 , σ 2 ) variables, then E [max i | (cid:15) i | ] ≤ σ (cid:112) 2 log(2 d ) .
Proof.
Deﬁne Z =
[max i | (cid:15) i | ] .
For any t > 0 , by Jensen’ inequality, we have
[Equation]
where the second inequality uses the deﬁnition of normal distribution and Φ is the cumulative distribution function of the standard normal distribution.
Taking log s on both sides and dividing by t , we get
[Equation]
Choosing t = (cid:112)
[Equation]
σ leads to the desired result,
[Equation]
We now return to the proof of Theorem 3.
Proof.
For any RNNs with weight matrices θ =
( U, W, V ) satisfying || V T || 1 ≤ β V , || W T || 1 ≤ β W ,
||
U T || 1 ≤ β U , we have
[Equation]
where the ﬁrst equality is due to the fact that the input x and noise (cid:15) are independent, the ﬁrst inequality uses Jensen’s inequality and the last inequality follows from Lemma 11.
The inequality above can be rewritten as follows.
[Equation]
For the ﬁrst term in the right hand side of the above inequality, by Theorem 2, with probability at least 1 − δ , the following holds:
[Equation]
[Table]
[Equation]
Combining the above two inequalities together leads to
[Equation]
where the ﬁrst inequality makes use of the fact that 1 u ≤ Φ α ( u ) .
Therefore, the desired result can be immediately obtained by substituting E (cid:15) ||
(cid:15)
i || ∞ with σ (cid:15) (cid:112) 2 log(2 d ) according to Lemma 12.
D S UPPLEMENTARY FIGURES
Figure 2 shows the behavior of generalization error for RNNs as the standard derivation of noise σ (cid:15) varies for the sequence length L = 200 and 300 trained on IMDB dataset.
As in the body, increasing σ (cid:15) will ﬁrst improve the generalization error and then, after a certain point, harm the performance of RNNs.