T HE R ECURRENT N EURAL T ANGENT K ERNEL
A BSTRACT
The study of deep neural networks (DNNs) in the inﬁnite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization.
One key DNN architecture remains to be kernelized, namely, the recurrent neural network (RNN).
In this paper we introduce and study the Recurrent Neural Tangent Kernel (RNTK), which provides new insights into the behavior of overparametrized RNNs.
A key property of the RNTK should greatly beneﬁt practitioners is its ability to compare inputs of different length.
To this end, we characterize how the RNTK weights different time steps to form its output under different initialization parameters and nonlinearity choices.
A synthetic and 56 real-world data experiments demonstrate that the RNTK offers signiﬁcant performance gains over other kernels, including standard NTKs, across a wide array of data sets.
I NTRODUCTION
The overparameterization of modern deep neural networks (DNNs) has resulted in not only remarkably good generalization performance on unseen data (Novak et al., 2018; Neyshabur et al., 2019; Belkin et al., 2019) but also guarantees that gradient descent learning can ﬁnd the global minimum of their highly nonconvex loss functions (Du et al., 2019b; Allen-Zhu et al., 2019b;a; Zou et al., 2018;
Arora et al., 2019b).
From these successes, a natural question arises: What happens when we take overparameterization to the limit by allowing the width of a DNN’s hidden layers to go to inﬁnity?
Surprisingly, the analysis of such an (impractical) DNN becomes analytically tractable.
Indeed, recent work has shown that the training dynamics of (inﬁnite-width) DNNs under gradient ﬂow is captured by a constant kernel called the Neural Tangent Kernel (NTK) that evolves according to a linear ordinary differential equation (ODE) (Jacot et al., 2018; Lee et al., 2019; Arora et al., 2019a).
Every DNN architecture and parameter initialization produces a distinct NTK.
The original NTK was derived from the Multilayer Perceptron (MLP)(Jacot et al., 2018) and was soon followed by kernels derived from Convolutional Neural Networks (CNTK) (Arora et al., 2019a; Yang, 2019a), Residual
DNNs (Huang et al., 2020), and Graph Convolutional Neural Networks (GNTK) (Du et al., 2019a).
In (Yang, 2020a), a general strategy to obtain the NTK of any architecture is provided.
In this paper, we extend the NTK concept to the important class of overparametrized Recurrent
Neural Networks (RNNs) , a fundamental DNN architecture for processing sequential data.
We show that RNN in its inﬁnite-width limit converges to a kernel that we dub the Recurrent Neural Tangent
Kernel (RNTK).
The RNTK provides high performance for various machine learning tasks, and an analysis of the properties of the kernel provides useful insights into the behavior of RNNs in the following overparametrized regime.
In particular, we derive and study the RNTK to answer the following theoretical questions:
Q: Can the RNTK extract long-term dependencies between two data sequences?
RNNs are known to underperform at learning long-term dependencies due to the gradient vanishing or exploding (Bengio et al., 1994).
Attempted ameliorations have included orthogonal weights (Arjovsky et al., 2016;
Jing et al., 2017; Henaff et al., 2016) and gating such as in Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Unit (GRU) (Cho et al., 2014) RNNs.
We demonstrate that the RNTK can detect long-term dependencies with proper initialization of the hyperparameters
, and moreover, we show how the dependencies are extracted through time via different hyperparameter choices.
Q: Do the recurrent weights of the RNTK reduce its representation power compared to other
NTKs?
An attractive property of an RNN that is shared by the RNTK is that it can deal with sequences of different lengths via weight sharing through time.
This enables the reduction of the number of learnable parameters and thus more stable training at the cost of reduced representation power.
We prove the surprising fact that employing tied vs. untied weights in an RNN does not impact the analytical form of the RNTK.
Q: Does the RNTK generalize well?
A recent study has revealed that the use of an SVM classiﬁer with the NTK, CNTK, and GNTK kernels outperforms other classical kernel-based classiﬁers and trained ﬁnite DNNs on small data sets (typically fewer than 5000 training samples)
(Lee et al., 2020;
Arora et al., 2019a; 2020; Du et al., 2019a).
We extend these results to RNTKs to demonstrate that the RNTK outperforms a variety of classic kernels, NTKs and ﬁnite RNNs for time series data sets in both classiﬁcation and regression tasks.
Carefully designed experiments with data of varying lengths demonstrate that the RNTK’s performance accelerates beyond other techniques as the difference in lengths increases.
Those results extend the empirical observations from (Arora et al., 2019a; 2020; Du et al., 2019a; Lee et al., 2020) into ﬁnite DNNs, NTK, CNTK, and GNTK comparisons by observing that their performance-wise ranking depends on the employed DNN architecture.
We summarize our contributions as follows:
[C1]
We derive the analytical form for the RNTK of an overparametrized RNN at initialization using rectiﬁed linear unit (ReLU) and error function (erf) nonlinearities for arbitrary data lengths and number of layers (Section 3.1).
[C2] We prove that the RNTK remains constant during (overparametrized) RNN training and that the dynamics of training are simpliﬁed to a set of ordinary differential equations (ODEs) (Section 3.2).
[C
] When the input data sequences are of equal length, we show that the RNTKs of weight-tied and weight-untied RNNs converge to the same RNTK (Section 3.3).
[C4] Leveraging our analytical formulation of the RNTK, we empirically demonstrate how correlations between data at different times are weighted by the function learned by an RNN for different sets of hyperparameters.
We also offer practical suggestions for choosing the RNN hyperparameters for deep information propagation through time (Section 3.4).
[C5]
We demonstrate that the RNTK is eminently practical by showing its superiority over classical kernels, NTKs, and ﬁnite RNNs in exhaustive experiments on time-series classiﬁcation and regression with both synthetic and 56 real-world data sets (Section 4).
B ACKGROUND AND R ELATED W ORK
Notation.
We denote [ n ] = { 1 , . . . , n } , and I d the identity matrix of size d .
[ A ] i,j represents the ( i, j ) -th entry of a matrix, and similarly [ a ] i represents the i -th entry of a vector.
We use φ ( · ) :
R → R to represent the activation function that acts coordinate wise on a vector and φ (cid:48) to denote its derivative.
We will often use the rectiﬁed linear unit (ReLU) φ ( x ) = max(0 , x ) and error function (erf) φ ( x ) = 2 √ π (cid:82) x 0 e − z 2 dz activation functions.
N ( µ , Σ ) represents the multidimensional
Gaussian distribution with the mean vector µ and the covariance matrix Σ .
Recurrent Neural Networks (RNNs).
Given an input sequence data x = { x t } Tt =1 of length T with data at time t , x t ∈ R m , a simple RNN (Elman, 1990) performs the following recursive computation at each layer (cid:96) and each time step t
[Equation]
) h ( (cid:96) − 1 ,t ) ( x ) + b ( (cid:96) ) , h ( (cid:96),t ) ( x ) = φ (cid:16) g ( (cid:96),t ) ( x ) (cid:17) , where W ( (cid:96) )
∈ R n × n , b ( (cid:96) )
∈ R n for (cid:96) ∈
[ L ] , U (1) ∈ R n × m and U ( (cid:96) )
∈ R n × n for (cid:96) ≥ 2 are the
RNN parameters.
g ( (cid:96),t ) ( x ) is the pre-activation vector at layer (cid:96) and time step t , and h ( (cid:96),t ) ( x ) is the after-activation (hidden state).
For the input layer (cid:96) = 0 , we deﬁne h (0 ,t ) ( x ) :
= x t .
h ( (cid:96), 0) ( x ) as the initial hidden state at layer (cid:96) that must be initialized to start the RNN recursive computation.
The output of an L -hidden layer RNN with linear read out layer is achieved via
[Equation]
where V ∈ R d × n
Figure 1 visualizes an RNN unrolled through time.
h (2 , 2) h
(2 , 1) ( x ) h (2 , 3) ( x ) h (1 , 2) ( x ) h (1 , 1) ( x ) h (1 , 3) ( x )
W (2) W (2)
W (1)
W (1) W (1) U (2)
U (2)
U (2) h (2 , 0) ( x ) h (1 , 0) ( x )
W (2)
W (1) xxx
U (1) U (1)
U (1)
Neural Tangent Kernel (NTK).
Let f θ ( x ) ∈ R
d be the output of a DNN with parameters θ .
For two input data sequences x and x (cid:48) , the NTK is deﬁned as (Jacot et al., 2018)
[Equation]
where f θ s and θ s are the network output and parameters during training at time s. 1 Let X and
Y be the set of training inputs and targets, (cid:96) ( (cid:98) y, y ) :
R d × R d → R + be the loss function, and
L = 1 |X| (cid:80) ( x , y ) ∈X×Y (cid:96) ( f θ s ( x ) , y ) be the the empirical loss.
The evolution of the parameters θ s and output of the network f θ s on a test input using gradient descent with inﬁnitesimal step size (a.k.a gradient ﬂow) with learning rate η is given by
[Equation]
[Equation]
Generally, (cid:98) Θ s ( x , x (cid:48) ) , hereafter referred to as the empirical NTK, changes over time during training, making the analysis of the training dynamics difﬁcult.
When f θ s corresponds to an inﬁnite-width
MLP, (Jacot et al., 2018) showed that (cid:98) Θ s ( x , x (cid:48) )
converges to a limiting kernel at initialization and stays constant during training, i.e.,
[Equation]
which is equivalent to replacing the outputs of the DNN by their ﬁrst-order Taylor expansion in the parameter space (Lee et al., 2019).
With a mean-square error (MSE) loss function, the training dynamics in (1) and (2) simplify to a set of linear ODEs, which coincides with the training dynamics of kernel ridge regression with respect to the NTK when the ridge term goes to zero.
A nonzero ridge regularization can be conjured up by adding a regularization term λ 2 2 (cid:107) θ s − θ 0 (cid:107) 22 to the empirical loss (Hu et al., 2020).
T HE R ECURRENT N EURAL T ANGENT K ERNEL
We are now ready to derive the RNTK.
We ﬁrst prove the convergence of an RNN at initialization to the RNTK in the inﬁnite-width limit and discuss various insights it provides.
We then derive the convergence of an RNN after training to the RNTK.
Finally, we analyze the effects of various hyperparameter choices on the RNTK.
Proofs of all of our results are provided in the Appendices.
3. RNTK FOR AN I NFINITE -W IDTH RNN AT I NITIALIZATION
First we specify the following parameter initialization scheme that follows previous work on NTKs (Jacot et al., 2018), which is crucial to our convergence results:
[Equation]
where
[Equation]
We will refer to (3) and (4) as the NTK initialization .
The choices of the hyperparameters σ w , σ u , σ v and σ b can signiﬁcantly impact RNN performance
[Equation]
and we discuss them in detail in Section
For the initial (at time t = 0 ) hidden state at each layer (cid:96) , we set h ( (cid:96), 0) ( x ) to an i.i.d.
copy of
N (0 , σ h ) (Wang et al., 2018) .
For convenience, we collect all of the learnable parameters of the
RNN into θ = vect (cid:2) {{ W ( (cid:96) ) , U ( (cid:96) ) , b ( (cid:96) ) } L(cid:96) =
1 , V } (cid:3) .
The derivation of the RNTK at initialization is based on the correspondence between Gaussian initialized, inﬁnite-width DNNs and Gaussian Processes (GPs), known as the DNN-GP.
In this setting every coordinate of the DNN output tends to a GP as the number of units/neurons in the hidden layer (its width) goes to inﬁnity.
The corresponding DNN-GP kernel is computed as
[Equation]
First introduced for a single-layer, fully-connected neural network by (Neal, 1995), recent works on
NTKs have extended the results for various DNN architectures (Lee et al., 2018; Duvenaud et al., 2014; Novak et al., 2019; Garriga-Alonso et al., 2019; Yang, 2019b), where in addition to the output, all pre-activation layers of the DNN tends to a GPs in the inﬁnite-width limit.
In the case of RNNs, each coordinate of the RNN pre-activation g ( (cid:96),t ) ( x ) converges to a centered GP depending on the inputs with kernel
[Equation]
As per (Yang, 2019a), the gradients of random inﬁnite-width DNNs computed during backpropagation are also Gaussian distributed.
In the case of RNNs, every coordinate of the vector δ ( (cid:96),t ) ( x ) := √ n (cid:0) ∇ g ( (cid:96),t ) ( x ) f θ ( x ) (cid:1) converges to a GP with kernel
[Equation]
Both convergences occur independently of the coordinate index i and for inputs of possibly different lengths, i.e., T (cid:54) =
T (cid:48) .
With (6) and (7), we now prove that an inﬁnite-width RNN at initialization converges to the limiting RNTK.
Theorem 1 Let x and x (cid:48) be two data sequences of potentially different lengths T and T (cid:48) , respectively.
Without loss of generality, assume that T ≤ T (cid:48) , and let τ := T (cid:48) − T .
Let n be the number of units in the hidden layers, the empirical RNTK for an L -layer RNN with NTK initialization converges to the following limiting kernel as n → ∞
[Equation]
where
[Equation]
9) with K ( x , x (cid:48) )
, Σ ( (cid:96),t,t + τ ) ( x , x (cid:48) ) , and Π ( (cid:96),t,t + τ ) ( x , x (cid:48) ) deﬁned in (5)–(7).
Remarks.
Theorem 1 holds generally for any two data sequences, including different lengths ones.
This highlights the RNTK’s ability to produce a similarity measure Θ( x , x (cid:48) ) even if the inputs are of different lengths, without resorting to heuristics such as zero padding the inputs to the to the max length of both sequences.
Dealing with data of different length is in sharp contrast to common kernels such as the classical radial basis functions, polynomial kernels, and current NTKs.
We showcase this capability below in Section 4.
To visualize Theorem 1, we plot in the left plot in Figure 2 the convergence of a single layer, sufﬁciently wide RNN to its RNTK with the two simple inputs x = { 1 , − 1 , 1 } of length 3 and x (cid:48)
= { cos( α ) , sin( α ) } of length 2, where α = [0 , 2 π ] .
For an RNN with a sufﬁciently large hidden state ( n = 1000 ), we see clearly that it converges to the RNTK ( n = ∞ ).
RNTK Example for a Single-Layer RNN.
We present a concrete example of Theorem 1 by showing how to recursively compute the RNTK for a single-layer RNN; thus we drop the layer index for notational simplicity.
We compute and display the RNTK for the general case of a multilayer RNN in Appendix B.3 .
To compute the RNTK Θ ( T,T (cid:48) )
( x , x (cid:48) ) , we need to compute the GP
[Figure]
( x , x (cid:48) )        0.
α l og ( (cid:107) Θ − (cid:98) Θ 0 Θ (cid:107) ) 2.
[Figure]
kernels Σ ( t,t + τ ) ( x , x (cid:48) ) and Π ( t,t + τ ) ( x , x (cid:48) ) .
We ﬁrst deﬁne the operator V φ (cid:2) K (cid:3) that depends on the nonlinearity φ ( · ) and a positive semi-deﬁnite matrix K ∈ R 2 ×
[Equation]
Following (Yang, 2019a), we obtain the analytical recursive formula for the GP kernel Σ ( t,t + τ ) ( x , x (cid:48) ) for a single layer RNN as
[Equation]
Similarly
[Equation]
we obtain the analytical recursive formula for the GP kernel Π ( [Equation] [Equation] ( x , x
[Equation]
) as Π
[Equation]
T [Equation] [Equation] For φ = ReLU and φ = erf , we provide analytical expressions for V φ (cid:2) K (cid:3) and V φ (cid:48) (cid:2) K (cid:3) in Appendix
B.5.
These yield an explicit formula for the RNTK that enables fast and point-wise kernel evaluations.
For other activation functions, one can apply the Monte Carlo approximation to obtain V φ (cid:2) K (cid:3) and
V φ (cid:48) (cid:2) K (cid:3) (Novak et al., 2019).
3. RNTK FOR AN I NFINITE -W IDTH RNN DURING T RAINING
We prove that an inﬁnitely-wide RNN, not only at initialization but also during gradient descent training, converges to the limiting RNTK at initialization.
Theorem 2 Let n be the number of units of each RNN’s layer.
Assume that Θ( X , X ) is positive deﬁnite on X such that λ min (Θ( X , X )) > 0 .
Let η ∗ :
= 2 (cid:0) λ min (Θ( X , X ))
+ λ max (Θ( X , X ) (cid:1)(cid:1) − 1 .
For an L -layer RNN with NTK initialization as in (3), (4) trained under gradient ﬂow (recall (1) and (2)) with η < η ∗ , we have with high probability
[Equation]
s
[Equation]
s ( t ) m a x t s ( t ) s ( t ) m a x t s ( t ) t t t t
Remarks.
Theorem 2 states that the training dynamics of an RNN in the inﬁnite-width limit as in (1), (2) are governed by the RNTK derived from the RNN at its initialization.
Intuitively, this is due to the NTK initialization (3), (4) which positions the parameters near a local minima, thus minimizing the amount of update that needs to be applied to the weights to obtain the ﬁnal parameters.
3. RNTK FOR AN I NFINITE -W IDTH RNN W ITHOUT W EIGHT S HARING
We prove that, in the inﬁnite-width limit, an RNN without weight sharing (untied weights), i.e., using independent new weights W ( (cid:96),t ) , U ( (cid:96),t ) and b ( (cid:96),t ) at each time step t , converges to the same
RNTK as an RNN with weight sharing (tied weights).
First, recall that it is a common practice to use weight-tied RNNs, i.e., in layer (cid:96) , the weights W ( (cid:96) ) , U ( (cid:96) ) and b ( (cid:96) ) are the same across all time steps t .
This practice conserves memory and reduces the number of learnable parameters.
We demonstrate that, when using untied-weights, the RNTK formula remains unchanged.
Theorem 3 For inputs of the same length, an RNN with untied weights converges to the same RNTK as an RNN with tied weights in the inﬁnite-width ( n → ∞ ) regime.
Remarks.
Theorem 3 implies that weight-tied and weight-untied RNNs have similar behaviors in the inﬁnite-width limit.
It also suggests that existing results on the simpler, weight-untied RNN setting may be applicable for the more general, weight-tied RNN.
The plot on the right side of Figureempirically demonstrates the convergence of both the weight-tied and weight-untied RNNs to the
RNTK with increasing hidden layer size n ; moreover, the convergence rates are similar.
3. I NSIGHTS INTO THE R OLES OF THE RNTK’ S H YPERPARAMETERS
Our analytical form for the RNTK is fully determined by a small number of hyperparameters, which contains the various weight variances collected into S
= { σ w , σ u , σ b , σ h } and the activation function.
In standard supervised-learning settings, one often performs cross-validation to select the hyperparameters.
However, since kernel methods become computationally intractable for large datasets, we seek a more computationally friendly alternative to cross-validation.
Here we conduct a novel exploratory analysis that provides new insights into the impact of the RNTK hyperparameters on the RNTK output and suggests a simple method to select them a priori in a deliberate manner.
To visualize the role of the RNTK hyperparameters, we introduce the sensitivity s ( t ) of the RNTK of two input sequences x and x (cid:48) with respect to the input x t at time t
[Equation]
[Equation] [Table] Here, s ( t ) indicates how sensitive the RNTK is to the data at time t , i.e., x t , in presence of another data sequence x (cid:48) .
Intuitively, large/small s ( t ) indicates that the RNTK is relatively sensitive/insensitive to the input x t at time t .
The sensitivity is crucial to understanding to which extent the RNTK prediction is impacted by the input at each time step.
In the case where some time indices have a small sensitivity, then any input variation in those corresponding times will not alter the RNTK output and thus will produce a metric that is invariant to those changes.
This situation can be beneﬁcial or detrimental based on the task at hand.
Ideally, and in the absence of prior knowledge on the data, one should aim to have a roughly constant sensitivity across time in order to treat all time steps equally in the RNTK input comparison.
Figure 3 plots the normalized sensitivity s ( t ) / max t ( s ( t )) for two data sequences of the same length
T = 100 , with s ( t ) computed numerically for x t , x (cid:48) t ∼ N (0 , 1) .
We repeated the experimentstimes; the mean of the sensitivity is shown in Figure 3.
Each of the plots shows the changes of parameters S ReLU = {√ 2 , 1 , 0 , 0 } for φ = ReLU and S erf = { 1 , 0 . 01 , 0 .
for φ = erf .
From Figure 3 we ﬁrst observe that both ReLU and erf show similar per time step sensitivity measure s ( t ) behavior around the hyperparameters S ReLU and S erf .
If one varies any of the weight variance parameters, the sensitivity exhibits a wide range of behavior, and in particular with erf .
We observe that σ w has a major inﬂuence on s ( t ) .
For ReLU , a small decrease/increase in σ w can lead to over-sensitivity of the RNTK to data at the last/ﬁrst times steps, whereas for erf , any changes in σ w leads to over-sensitivity to the last time steps.
Another notable observation is the importance of σ h , which is usually set to zero for RNNs.
(Wang et al., 2018) showed that a non-zero σ h acts as a regularization that improves the performance of RNNs with the ReLU nonlinearity.
From the sensitivity perspective, a non-zero σ h results in reducing the importance of the ﬁrst time steps of the input.
We also see the same behavior in erf , but with stronger changes as σ h increases.
Hence whenever one aims at reinforcing the input pairwise comparisons, such parameters should be favored.
This sensitivity analysis provides a practical tool for RNTK hyperparameter tuning.
In the absence of knowledge about the data, hyperparameters should be chosen to produce the least time varying sensitivity.
If given a priori knowledge, hyperparameters can be selected that direct the RNTK to the desired time-steps.
E XPERIMENTS
We now empirically validate the performance of the RNTK compared to classic kernels, NTKs, and trained RNNs on both classiﬁcation and regression tasks using a large number of time series data sets.
Of particular interest is the capability of the RNTK to offer high performance even on inputs of different lengths.
Time Series Classiﬁcation.
The ﬁrst set of experiments considers time series inputs of the same lengths from 56 datasets in the UCR time-series classiﬁcation data repository (Dau et al., 2019).
We restrict ourselves to selected data sets with fewer than 1000 training samples and fewer than 1000 time steps ( T ) as kernel methods become rapidly intractable for larger datasets.
We compare the RNTK with a variety of other kernels, including the Radial Basis Kernel (RBF), polynomial kernel, and NTK (Jacot et al., 2018), as well as ﬁnite RNNs with Gaussian, identity (Le et al., 2015) initialization, and GRU (Cho et al., 2014).
We use φ = ReLU for both the RNTKs and NTKs.
For each kernel, we train a C-SVM (Chang & Lin, 2011) classiﬁer, and for each ﬁnite RNN we use gradient descent training.
For model hyperparameter tuning, we use 10-fold cross-validation.
Details on the data sets and experimental setup are available in Appendix A.1. [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] We summarize the classiﬁcation results over all 56 datasets in Table 1; detailed results on each data set is available in Appendix A.2.
We see that the RNTK outperforms not only the classical kernels but also the NTK and trained RNNs in all metrics.
The results demonstrate the ability of RNTK to provide increased performances compare to various other methods (kernels and RNNs).
The superior performance of RNTK compared to other kernels, including NTK, can be explained by the internal recurrent mechanism present in RNTK, allowing time-series adapted sample comparison.
In addition,
RNTK also outperforms RNN and GRU.
As the datasets we consider are relative small in size, ﬁnite
RNNs and GRUs that typically require large amount of data to succeed do not perform well in our setting.
An interesting future direction would be to compare RNTK to RNN/GRU on larger datasets.
Time Series Regression.
We now validate the performance of the RNTK on time series inputs of different lengths on both synthetic data and real data.
For both scenarios, the target is to predict the next time-step observation of the randomly extracted windows of different length using kernel ridge regression.
We compare the RNTK to other kernels, the RBF and polynomial kernels and the NTK.
We also compare our results with a data independent predictor that requires no training, that is simply to predict the next time step with previous time step (PTS).
For the synthetic data experiment, we simulate 1000 samples of one period of a sinusoid and add white
Gaussian noise with default σ n = 0 .
From this ﬁxed data, we extract training set size N train =segments of uniform random lengths in the range of [ T ﬁxed , T ﬁxed + T var ] with T ﬁxed = 10 .
We use standard kernel ridge regression for this task.
The test set is comprised of N test = 5000 obtained from other randomly extracted segments, again of varying lengths.
For the real data, we use 975 days of the Google stock value in the years 2014–2018.
As in the simulated signal setup above, we extract
N train segments of different lengths from the ﬁrst 700 days and test on the N test segments from days 701 to 975.
Details of the experiment are available in Appendix A.2.
We report the predicted signal-to-noise ratio (SNR) for both datasets in Figures 4a and 4c for various values of T var .
We vary the noise level and training set size for ﬁxed T var = 10 in Figures 4b and 4d.
As we see from Figures 4a and 4c, the RNTK offers substantial performance gains compared to the other kernels, due to its ability to naturally deal with variable length inputs.
Moreover, the performance gap increases with the amount of length variation of the inputs T var .
Figure 4d demonstrates that, unlike the other methods, the RNTK maintains its performance even when the training set is small.
Finally, Figure 4c demonstrates that the impact of noise in the data on the regression performance is roughly the same for all models but becomes more important for RNTK with a large σ n ; this might be attributed to the recurrent structure of the model allowing for a time propagation and ampliﬁcation of the noise for very low SNR.
These experiments demonstrate the distinctive advantages of the RNTK over classical kernels, and NTKs for input data sequences of varying lengths.
In the case of PTS, we expect the predictor to outperform kernel methods when learning from the training samples is hard, due to noise in the data or small training size which can lead to over ﬁtting.
In Figure 4a RNTK and Polynomial kernels outperforms PTS for all values of T var , but for larger
T var , NTK and RBF under perform PTS due to the increasing detrimental effect of zero padding.
For the Google stock value, we see a superior performance of PTS with respect to all other kernel methods due to the nature of those data heavily relying on close past data.
However, RNTK is able to reduce the effect of over-ﬁtting, and provide the closest results to PTS among all kernel methods we employed, with increasing performance as the number of training samples increase.
C ONCLUSIONS
In this paper, we have derived the RNTK based on the architecture of a simple RNN.
We have proved that, at initialization, after training, and without weight sharing, any simple RNN converges to the same RNTK.
This convergence provides new insights into the behavior of inﬁnite-width RNNs, including how they process different-length inputs, their training dynamics, and the sensitivity of their output at every time step to different nonlinearities and initializations.
We have highlighted the RNTK’s practical utility by demonstrating its superior performance on time series classiﬁcation and regression compared to a range of classical kernels, the NTK, and trained RNNs.
There are many avenues for future research, including developing RNTKs for gated RNNs such as the
LSTM (Hochreiter & Schmidhuber, 1997) and investigating which of our theoretical insights extend to ﬁnite RNNs.
A CKNOWLEDGMENTS
This work was supported by NSF grants CCF-1911094, IIS-1838177, and IIS-1730574; ONR grants
N00014-18-12571, N00014-20-1-2787, and N00014-20-1-2534; AFOSR grant FA9550-18-1-0478; and a Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047.
A E XPERIMENT D ETAILS A. T IME SERIES CLASSIFICATION
Kernel methods settings.
We used RNTK, RBF, polynomial and NTK (Jacot et al., 2018).
For data pre-processing, we normalized the norm of each x to 1.
For training we used C-SVM in LIBSVM library (Chang & Lin, 2011) and for hyperparameter selection we performed 10-fold validation for splitting the training data into 90% training set and 10% validation test.
We then choose the best performing set of hyperparameters on all the validation sets, retrain the models with the best set of hyperparameters on the entire training data and ﬁnally report the performance on the unseen test data.
The performance of all kernels on each data set is shown in table 2.
For C-SVM we chose the cost function value
C ∈ { 0 . 01 , 0 . 1 , 1 , 10 , 100 } and for each kernel we used the following hyperparameter sets
• RNTK:
We only used single layer RNTK, we φ = ReLU and the following hyperparameter sets for the variances: σ w ∈ { 1 . 34 , 1 . 35 , 1 . 36 , 1 . 37 , 1 . 38 , 1 . 39 , 1 . 40 , 1 . 41 , 1 . 42 , √ 2 , 1 . 43 , 1 . 44 , 1 . 45 , 1 . 46 , 1 . 47 } σ u =σ b ∈ { 0 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 7 , 0 . 9 , 1 , 2 } σ h ∈ { 0 , 0 . 01 , 0 . 1 , 0 . 5 , 1 } • NTK: The formula for NTK of L -layer MLP (Jacot et al., 2018) for x , x
(cid:48) ∈
R m is:
[Equation]
and we used the following hyperparamters
[Equation] [Equation]
• RBF:
[Equation]
30 , 40 , 100 } • Polynomial:
[Equation]
[Table]
nsectEPGRegular 99.
96.79 98.39 Meat 93.
33.33 Lightning 78.
67.21 Lightning 61.
Car 83.
26.66 GunPoint 95.33 95.33 74.
80.66 Arrowhead 80.
83.43 80.57 74.86 37.71 Coffee  92.86 92.8642.
57.14 Trace
ECG       76 plane 98.
96.19 GunPointOldVersusYoung 98.
98.41 GunPointMaleVersusFemale 99.
99.68 GunPointAgeSpan 96.
95.56 FreezerRegularTrain 97.
86.59 SemgHandSubjectCh 84.
WormsTwoClass 62.
57.14 Earthquakes 74.
FiftyWords 68.
65.27 Beef
[Table] [Table]
Finite-width RNN settings.
We used 3 different RNNs.
The ﬁrst is a ReLU RNN with Gaussian initialization with the same NTK initialization scheme, where parameter variances are σ w = σ v = √ 2 , σ u = 1 and σ b = 0 .
The second is a ReLU RNN with identity initialization following (Le et al., 2015).
The third is a GRU (Cho et al., 2014) with uniform initialization.
All models are trained with
RMSProp algorithm for 200 epochs.
Early stopping is implemented when the validation set accuracy does not improve for 5 consecutive epochs.
We perform standard 5-fold cross validation.
For each RNN architecture we used hyperparamters of number of layer, number of hidden units and learning rate as
[Equation]
Metrics descriptions First, only in this paragraph, let i ∈ { 1 , 2 , ..., N } index a total of N datasets and j ∈ { 1 , 2 , ..., M } index a total of M classiﬁers.
Let y ij be the accuracy of the j -th classifer on the i -th dataset.
We reported results on 4 metrics: average accuracy (Acc. mean), P90, P95, PMA and Friedman Rank.
P90 and P95 is the fraction of datasets that the classiﬁer achieves at least 90% and 95% of the maximum achievable accuracy for each dataset, i.e., [Equation]
[Equation]
PMA is the accuracy of the classiﬁer on a dataset divided by the maximum achievable accuracy on that dataset, averaged over all datasets:
[Equation]
[Equation]
Friedman Rank (Fernández-Delgado et al., 2019)
ﬁrst ranks the accuracy of each classiﬁer on each dataset and then takes the average of the ranks for each classiﬁer over all datasets, i.e.,
[Equation]
) where r ij is the ranking of the j -th classiﬁer on the i -th dataset.
Note that a better classiﬁer achieves a lower Friedman Rank, Higher P/90 and PMA.
Remark.
In order to provide insight into the performance of RNTK in long time steps setting, we picked two datasets with more that 1000 times steps: SemgHandSubjectCh2 ( T = 1024 ) and
StarLightCurves ( T = 1024 ).
A. T IME S ERIES R EGRESSION
For time series regression, we used the 5-fold validation of training set and same hyperparamter sets for all kernels.
For training we kernel ridge regression with ridge term chosen form λ ∈ { 0 , 0 . 01 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 8 , 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 10 , 100 }
B P ROOFS FOR T HEOREMS 1 AND 3: RNTK C ONVERGENCE AT I NITIALIZATION B. P RELIMINARY : N ETSOR PROGRAMS
Calculation of NTK in any architecture relies on ﬁnding the GP kernels that correspond to each pre-activation and gradient layers at initialization.
For feedforward neural networks with n 1 , . . . , n L number of neurons (channels in CNNs) at each layer the form of this GP kernels can be calculated via taking the limit of n 1 , . . .
, n L sequentially one by one.
The proof is given by induction, where by conditioning on the previous layers, each entry of the current layer is sum of inﬁnite i.i.d Gaussian random variables, and based on Central Limit Theorem (CLT), it becomes a Gaussian process with kernel calculated based on the previous layers.
Since the ﬁrst layer is an afﬁne transformation of input with Gaussian weights, it is a Gaussian process and the proof is completed.
See (Lee et al., 2018; Duvenaud et al., 2014; Novak et al., 2019; Garriga-Alonso et al., 2019) for a formal treatment.
However, due to weight-sharing, sequential limit is not possible and condoning on previous layers does not result in i.i.d.
weights.
Hence the aforementioned arguments break.
To deal with it, in (Yang, 2019a) a proof using Gaussian conditioning trick (Bolthausen, 2014) is presented which allows use of recurrent weights in a network.
More precisely, it has been demonstrated than neural networks (without batch normalization) can be expressed and a series of matrix multiplication and (piece wise) nonlinearity application, generally referred as Netsor programs .
It has been shown that any architecture that can be expressed as Netsor programs that converge to GPs as width goes to inﬁnity in the same rate, which a general rule to obtain the GP kernels.
For completeness of this paper, we brieﬂy restate the results from (Yang, 2019a) which we will use later for calculation derivation of
RNTK.
There are 3 types of variables in Netsor programs ; A -vars, G -vars and H -vars.
A -vars are matrices and vectors with i.i.d Gaussian entries, G -vars are vectors introduced by multiplication of a vector by an A -var and H -vars are vectors after coordinate wise nonlinearities is applied to G -vars.
Generally,
G -vars can be thought of as pre-activation layers which are asymptotically treated as a Gaussian distributed vectors, H -vars as after-activation layers and A -vars are the weights.
Since in neural networks inputs are immediately multiplied by a weight matrix, it can be thought of as an G -var, namely g in .
Generally Netsor programs supports G -vars with different dimension, however the asymptotic behavior of a neural networks described by Netsor programs does not change under this degree of freedom, as long as they go to inﬁnity at the same rate.
For simplicity, let the G -vars and
H -vars have the same dimension n since the network of interest is RNN and all pre-activation layers have the same dimension.
We introduce the Netsor programs under this simpliﬁcation.
To produce the output of a neural network, Netsor programs receive a set of G -vars and A -vars as input, and new variables are produced sequentially using the three following operators:
• Matmul : multiplication of an A -var:
A with an H -var: h , which produce a new G -var, g .
[Equation]
• Lincomp : Linear combination of G -vars, g i , 1 ≤ i ≤ k , with coefﬁcients a i ∈ R
[Equation]
≤ i ≤ k which produce of new G -var:
[Equation]
• Nonlin : creating a new H -var, h , by using a nonlinear function φ : R k
→ R that act coordinate wise on a set of G -vars, g i , 1 ≤ i ≤ k :
[Equation]
Any output of the neural network y ∈ R should be expressed as inner product of a new A -var which has not been used anywhere else in previous computations and an H -var:
[Equation]
Any other output can be produced by another v (cid:48) and h (cid:48)
(possibility the same h or v ).
It is assumed that each entry of any A -var : A ∈ R n × n in the netsor programs computations is drawn from N (0 , σ 2 a n ) and the input G -vars are Gaussian distributed.
The collection of a speciﬁc entry of all
G -vars of in the netsor program converges in probability to a Gaussian vector { [ g 1 ] i , . . .
, [ g k ] i } ∼
N ( µ , Σ ) for all i ∈ [ n ] as n goes to inﬁnity.
Let µ ( g ) :
=
E (cid:2)
[ g ] i (cid:3) be the mean of a G -var and Σ( g , g (cid:48) ) :
= E (cid:2) [ g ] i · [ g (cid:48) ]
i (cid:3) be the covariance between any two G -vars.
The general rule for µ ( g ) is given by the following equations:
[Equation]
input
[Equation]
otherwise (27)
For g and g (cid:48) , let G = { g 1 , . .
, g r } be the set of G -vars that has been introduced before g and g (cid:48) with distribution N ( µ G , Σ G ) , where Σ G ∈ R |G|×|G| containing the pairwise covariances between the G -vars.
Σ( g , g (cid:48) ) is calculated via the following rules:
[Equation]
if g and g (cid:48)
[Equation]
inputs
[Equation]
otherwise (28)
Where h =
ϕ
( g 1 , . . .
, g r ) and h (cid:48) =
¯ ϕ
( g 1 , . . .
, g r ) are functions of G -vars in G from possibly different nonlinearities.
This set of rules presents a recursive method for calculating the GP kernels in a network where the recursive formula starts from data dependent quantities Σ in and µ in which are given.
All the above results holds when the nonlinearities are bounded uniformly by e ( cx 2 − α ) for some α > 0 and when their derivatives exist.
Standard vs.
NTK
initialization.
The common practice (which netsor programs uses) is to initialize DNNs weights
[ A ] i,j with N (0 , σ a √ n ) (known as standard initialization) where generally n is the number of units in the previous layer.
In this paper we have used a different parameterization scheme as used in (Jacot et al., 2018) and we factor the standard deviation as shown in 3 and initialize weights with standard standard Gaussian.
This approach does not change the the forward computation of DNN, but normalizes the backward computation (when computing the gradients) by factor 1 n , otherwise RNTK will be scales by n .
However this problem can be solved by scaling the step size byn and there is no difference between NTK and standard initialization (Lee et al., 2019).
B. P ROOF FOR T HEOREM 1: S INGLE LAYER C ASE
We ﬁrst derive the RNTK in a simpler setting, i.e., a single layer and single output RNN.
We then generalize the results to multi-layer and multi-output RNNs.
We drop the layer index (cid:96) to simplify notation.
From 3 and 4, the forward pass for computing the output under NTK initialization for each input x = { x t } Tt =1 is given by:
[Equation] [Equation]
[Equation]
Note that (29), (30) and (31) use all the introduced operators introduced in 24, 25 and 26 given input variables W , { U x t } Tt =1 , b , v
and h (0) ( x ) .
First, we compute the kernels of forward pass Σ ( t,t (cid:48) )
( x , x (cid:48) ) and backward pass Π ( t,t (cid:48) )
( x , x (cid:48) ) introduced in (6) and (7) for two input x and x (cid:48) .
Note that based on (27) the mean of all variables is zero since the inputs are all zero mean.
In the forward pass for the intermediate layers we have: Σ ( t,t (cid:48) ) ( x , x (cid:48)
[Equation]
We have used the second and third rule in (28) to expand the formula, We have also used the ﬁrst and ﬁfth rule to set the cross term to zero, i.e.,
[Equation]
For the non-zero terms we have
[Equation]
which can be achieved by straight forward computation.
If t (cid:54) = 1 and t (cid:48) (cid:54) = 1 , by using the forth rule in (28) we have Σ
[Equation]
With K ( t,t (cid:48) )
( x , x (cid:48) ) deﬁned in (16).
Otherwise, it will be zero by the ﬁfth rule (if t or t = 1) .
Here the set of previously introduced G -vars is G = [Equation]
} , U
[Equation] [Equation]
but the dependency is only on the last layer G -vars, ϕ ( { g : g ∈ G} ) = φ ( g ( t
[Equation] 1) ( x )) , ¯ ϕ (( { g : g ∈ G} )) =
φ ( g ( t (cid:48) − 1) ( x (cid:48) )) , leading the calculation to the operator deﬁned in (10).
As a result
[Equation]
To complete the recursive formula, using the same procedure for the ﬁrst layers we have
[Equation]
The output GP kernel is calculated via
[Equation]
The calculation of the gradient vectors δ ( t ) ( x ) = √ n (cid:0) ∇ g ( t ) ( x ) [Equation] θ ( x ) (cid:1) in the backward pass is given by
[Equation]
To calculate the backward pass kernels [Equation] we rely on the following Corollary from (Yang, 2020b)
Corollary 1 In inﬁnitely wide neural networks weights used in calculation of back propagation gradients ( W (cid:62) ) is an i.i.d copy of weights used in forward propagation ( W ) as long as the last layer weight ( v ) is sampled independently from other parameters and has mean 0.
The immediate result of Corollary 1 is that g ( t ) ( x ) and δ ( t ) ( x ) are two independent Gaussian vector as their covariance is zero based on the ﬁfth rule in (28).
Using this result
[Equation]
we have:
[Equation]
If T
[Equation]
− t (cid:48)
=
T − t , then the the formula will lead to
[Equation]
Otherwise it will end to either of two cases for some t (cid:48)(cid:48) <
T or T (cid:48) and by the ﬁfth rule in (28) we have:
[Equation]
Without loss of generality
[Equation]
from now on assume T (cid:48) <
T and T (cid:48) − T = τ , the ﬁnal formula for computing the backward gradients becomes:
[Equation]
Now we have derived the single layer RNTK.
Recall that θ = Vect (cid:2) { W , U , b , v } (cid:3) contains all of the network’s learnable parameters.
As a result
[Equation]
we have:
[Equation]
As a result
[Equation]
, ∇ θ f θ ( x (cid:48) ) (cid:105) = (cid:28) ∂f θ ( x ) ∂ W , ∂f θ ( x (cid:48) )
∂ W (cid:29) + (cid:28) ∂f θ ( x ) ∂ U , ∂f θ ( x (cid:48) ) ∂ U (cid:29) + (cid:28)
∂f θ ( x ) ∂ b , ∂f θ ( x (cid:48) )
∂
b (cid:29) + (cid:28) ∂f θ ( x ) ∂ v , ∂f θ ( x (cid:48) )
∂ v (cid:29)
Where the gradients of output with respect to weights can be formulated as the following compact form:
[Equation]
As a result we have:
[Equation]
Remember that for any two G -var E
[[ g ] i [ g (cid:48) ] i ] is independent of index i .
Therefore,
[Equation]
[Equation]
Hence
[Equation]
by summing the above terms in the inﬁnite-width limit we get [Equation] ∇ [Equation]
f
[Equation]
x
[Equation]
→
[Equation] t (cid:48) = [Equation] T (cid:88) t = Π
[Equation]
[Equation]
x [Equation] )   + K ( x
[Equation]
(cid:48) ) .
Since Π ( t,t (cid:48) )
( x , x (cid:48) )
=
for t (cid:48) − t (cid:54) =
τ it is simpliﬁed to (cid:104)∇ θ f θ ( x ) , ∇ θ f θ ( x (cid:48) ) (cid:105) = (cid:32) T (cid:88) t
= Π ( t,t + τ ) ( x , x (cid:48) ) · Σ ( t,t + τ ) ( x (cid:48) , x (cid:48) ) (cid:33) + K ( x , x (cid:48) ) .
Multi-dimensional output.
For f θ ( x ) ∈ R d , the i -th output for i ∈ [ d ] is obtained via
[Equation]
where v i is independent of v j for i (cid:54)
=
j .
As a result, for The RNTK Θ( x , x (cid:48) )
∈ R
d × d for multidimensional output we have
[Equation]
For i =
j , the kernel is the same as computed in (32) and we denote it as
[Equation]
For i (cid:54) =
j , since v i is independent of v j , Π ( T,T (cid:48) )
( x , x [Equation] ) and all the backward pass gradients become zero, so
[Equation]
which gives us the following formula
[Equation]
This concludes the proof for Theorem 1 for single-layer case
[Equation]
B. P ROOF FOR T HEOREM 1: M ULTI -L AYER C ASE
Now we drive the RNTK for multi-layer RNTK.
We will only study single output case and the generalization to multi-dimensional case is identical as the single layer case.
The set of equations for calculation of the output of a L -layer RNN for x = { x t } Tt =1 are
[Equation]
The forward pass kernels for the ﬁrst layer is the same as calculated in B
[Equation]
For (cid:96) ≥ 2 we have:
[Equation]
where
[Equation]
and Σ in is deﬁned in (B.2).
For the ﬁrst ﬁrst time step we have:
[Equation]
And the output layer
[Equation]
Note that because of using new weights at each layer we get
[Equation]
Now we calculate the backward pass kernels in multi-layer RNTK.
The gradients at the last layer is calculated via
[Equation]
In the last hidden layer for different time steps we have
[Equation]
In the last time step for different hidden layers we have
[Equation]
At the end for the other layers we have
[Equation]
The recursive formula for the Π ( L,t,t (cid:48) )
( x , x (cid:48) ) is the same as the single layer, and it is non-zero for t (cid:48)
− t = T (cid:48) − T = τ .
As a result we have
[Equation]
Similarly by using the same course of arguments used in the single layer setting
[Equation]
for the last time step we have
[Equation]
For the other layers we have
[Equation]
For [Equation] = τ the recursion continues until it reaches Π ( L,T,t (cid:48)(cid:48) )
( x , x (cid:48) ) , t (cid:48)(cid:48) <
T (cid:48) or Π ( L,t (cid:48)(cid:48) ,T (cid:48) )
( x , x (cid:48) ) , t (cid:48)(cid:48) <
T and as a result based on (33) we get Π ( (cid:96),t,t (cid:48) )
( x , x (cid:48) )
=
t (cid:48) − t (cid:54) = τ (34)
For t (cid:48) − t = τ it leads to Π ( L,T,T (cid:48) )
( x , x (cid:48) ) and has a non-zero value.
Now we derive RNTK for multi-layer:
[Equation]
, where
[Equation]
θ ( x ) ∂ W ( (cid:96) ) , ∂f θ ( x (cid:48) )
∂ W ( (cid:96) )
(cid:29) =
T (cid:48) (cid:88) t (cid:48) =
T (cid:88) t
= (cid:18)n (cid:68) δ ( (cid:96),t ) ( x ) , δ ( (cid:96),t (cid:48) )
( x (cid:48) )
(cid:69)(cid:19) · (cid:18) ( σ (cid:96)w )n (cid:68) h
( (cid:96),t − 1) ( x ) , h ( (cid:96),t (cid:48) − 1)
( x (cid:48) ) (cid:69)(cid:19) (cid:28)
∂f θ ( x ) ∂ U ( (cid:96) ) , ∂f θ ( x (cid:48) )
∂ U ( (cid:96) )
(cid:29) =
T (cid:48) (cid:88) t (cid:48) =
T (cid:88) t
= (cid:18)n (cid:68) δ ( (cid:96),t ) ( x ) , δ ( (cid:96),t (cid:48) )
( x (cid:48) )
(cid:69)(cid:19) · (cid:18) ( σ (cid:96)u )m (cid:104) x t , x (cid:48) t (cid:48) (cid:105) (cid:19) (cid:96) =(cid:28)
∂f θ ( x ) ∂
U ( (cid:96) ) , ∂f θ ( x (cid:48) )
∂ U ( (cid:96) )
(cid:29) =
T (cid:48) (cid:88) t (cid:48) =
T (cid:88) t
= (cid:20) (cid:18)n (cid:68) δ ( (cid:96),t ) ( x ) , δ ( (cid:96),t (cid:48) ) ( x (cid:48) ) (cid:69)(cid:19) · (cid:18) ( σ (cid:96)u )n (cid:68) h ( (cid:96) − 1 ,t ) ( x ) , h ( (cid:96) − 1 ,t (cid:48) )
( x (cid:48) ) (cid:69)(cid:19) (cid:21) (cid:96) >(cid:28)
∂f θ ( x ) ∂
b ( (cid:96) ) , ∂f θ ( x (cid:48) )
∂ b ( (cid:96) ) (cid:29) = T (cid:48) (cid:88) t (cid:48) = T (cid:88) t
= (cid:18)n (cid:68) δ (
[Table]
,t ) ( x ) , δ (
[Table]
,t
[Table]
) ( x
[Table]
) (cid:69)(cid:19) · ( σ (cid:96)b )(cid:28) ∂f θ ( x ) ∂ v , ∂f θ ( x (cid:48) ) ∂ v (cid:29) = (cid:18) σ 2 v n (cid:68) h ( T ) ( x ) , h ( T (cid:48) ) ( x
[Table]
) (cid:69)(cid:19)
Summing up all the terms and replacing the inner product of vectors with their expectations we get
[Equation]

[Equation]
By (34), we can simplify to
[Equation]
For multi-dimensional output it becomes
[Equation]
This concludes the proof for Theorem 1 for the multi-layer case.
B. P ROOF FOR T HEOREM 3: W EIGHT -U NTIED RNTK
The architecture of a weight-untied single layer RNN is
[Equation]
Where we use new weights at each time step and we index it by time.
Like previous sections, we ﬁrst derive the forward pass kernels for two same length data x = { x t } Tt =1 , x = { x [Equation] t
[Equation]
} Tt (cid:48) = [Equation] Since we are using same weight at the same time step, Σ ( t,t ) ( x , x (cid:48) ) can be written as a function of the previous kernel, which is exactly as the weight-tied RNN.
However for different length, it becomes zero as a consequence of using different weights, unlike weight-tied which has non-zero value.
The kernel of the ﬁrst time step and output is also the same as weight-tied RNN.
For the gradients we have:
[Equation]
t
[Equation]
[Equation]
For t (cid:48)
=
t we have:
[Equation]
t,t ) ( x , x (cid:48) )
=
σ 2 v V φ (cid:48) (cid:2) K ( T +1 ,T + τ +1)
( x , x (cid:48) ) (cid:3) .
Due to using different weights for t (cid:54) =
t (cid:48) , we can immediately conclude that Π ( t,t (cid:48) )
( x , x (cid:48) )
= 0 .
This set of calculation is exactly the same as the weight-tied case when τ = T − T = 0 .
Finally, with θ = Vect (cid:2) {{ W ( t ) , U ( t ) , b ( t ) } Tt =1 , v } (cid:3) we have (cid:104)∇ θ f θ ( x ) , ∇ θ f θ ( x (cid:48) ) (cid:105) =
T (cid:88) t = (cid:28) ∂f θ ( x ) ∂ W ( t ) , ∂f θ ( x (cid:48) ) ∂ W ( t ) (cid:29) + T (cid:88) t = (cid:28) ∂f θ ( x ) ∂ U ( t ) , ∂f θ ( x (cid:48) ) ∂ U ( t ) (cid:29) +
T (cid:88) t = (cid:28) ∂f θ ( x ) ∂
b ( t ) , ∂f θ ( x (cid:48) )
∂ b ( t ) (cid:29) + (cid:28) ∂f θ ( x ) ∂ v , ∂f θ ( x (cid:48) )
∂ v , (cid:29) with (cid:28)
∂f θ ( x ) ∂ W ( t ) , [Equation] θ ( x (cid:48)
[Equation]
∂ W ( t ) (cid:29) [Equation]
(cid:18)n (cid:68)
[Equation] ( t [Equation] ( x
[Equation] [Equation]
( t [Equation] ( x (cid:48) ) (cid:69)(cid:19) · (cid:18)
[Equation]
n (cid:68) h ( t
[Equation]
[Equation] ( x )
[Equation]
h ( t [Equation] [Equation] ( x (cid:48)
[Equation] (cid:69)(cid:19) (cid:28) ∂f θ ( x ) ∂ U ( t ) , [Equation]
( x (cid:48) [Equation] U ( [Equation]
(cid:28) [Equation] [Equation] [Equation]
(cid:28) [Equation] ( x ) [Equation]
v
[Equation]
x
[Equation]
[Equation]
As a result we obtain
[Equation]
t =
Π ( t,t ) ( x , x (cid:48) ) ·
[Equation]
( t,t ) (
x (cid:48)
[Equation]
x
[Equation]
(cid:33)
+ K
[Equation]
x
[Equation]
x (cid:48) )
[Equation]
same as the weight-tied RNN when τ = 0 .
This concludes the proof for Theorem 3.
B. A NALYTICAL F ORMULA FOR V φ [ K ]
For any positive deﬁnite matrix K = (cid:20) K
K 3 K
K(cid:21) we have:
• φ = ReLU
(Cho & Saul, 2009)
[Equation]
where c = K 3 / √ K 1 K• φ = erf (Neal, 1995)
[Equation]
C P ROOF FOR T HEOREM 2: RNTK C ONVERGENCE AFTER T RAINING
To prove theorem 2, we use the strategy used in (Lee et al., 2019) which relies on the the local lipschitzness of the network Jacobian J ( θ, X ) = ∇ θ f θ ( x ) ∈ R |X|
d ×| θ | at initialization.
Deﬁnition 1 The Jacobian of a neural network is local lipschitz at NTK initialization ( θ 0 ∼ N (0 , 1)) if there is constant K > 0 for every C such that
[Equation]
where
[Equation]
Theorem 4 Assume that the network Jacobian is local lipschitz with high probability and the empirical NTK of the network converges in probability at initialization and it is positive deﬁnite over the input set.
For (cid:15) > 0 , there exists N such that for n > N when applying gradient ﬂow with η < 2 ( λ min (Θ( X , X ))
+ λ max (Θ( X , X ))
− 1 with probability at least (1 − (cid:15) )
we have:
[Equation]
Proof: See (Lee et al., 2019)
Theorem 4 holds for any network architecture and any cost function and it was used in (Lee et al., 2019) to show the stability of NTK for MLP during training.
Here we extend the results for RNTK by proving that the Jacobian of a multi-layer RNN under NTK initialization is local lipschitz with high probability.
To prove it, ﬁrst, we prove that for any two points θ, ˜ θ ∈ B ( θ 0 , R ) there exists constant K 1 such that
[Equation]
To prove (35) and (36) we use the following lemmas.
Lemma 1 Let A ∈ R n × m be a random matrix whose entries are independent standard normal random variables.
Then for every t ≥ 0 , with probability at least 1 − e ( − ct 2 ) for some constant c we have:
[Equation]
Lemma 2 Let a ∈ R n be a random vector whose entries are independent standard normal random variables.
Then for every t ≥ 0 , with probability at least 1 − e ( − ct 2 ) for some constant c we have:
[Equation]
Setting t = √ n for any θ ∈ R ( θ 0 , R ) .
With high probability, we get:
[Equation]
We also assume that there exists some ﬁnite constant C such that
[Equation]
The proof is obtained by induction.
From now on assume that all inequalities in (35) and (36) holds with some k for the previous layers.
We have
[Equation]
And the proof for (35) and (36) is completed by showing that the ﬁrst layer is bounded
[Equation]
For the gradient of ﬁrst layer we have
[Equation]
And similarly we have
[Equation]
For θ, ˜ θ ∈ B ( θ 0 , R ) we have
[Equation]
For gradients we have
[Equation]
And similarly using same techniques we have
[Equation]
As a result
[Equation]
there exists K 1 that is a function of σ w , σ u , σ b , L, T and the norm of the inputs.
Now we prove the local Lipchitzness of the Jacobian (cid:107)
[Equation]
( (cid:96),t − 1) ( x ) (cid:17) (cid:62) (cid:13)(cid:13)(cid:13)(cid:13)
F +n (cid:13)(cid:13)(cid:13)(cid:13) δ ( (cid:96),t ) ( x ) (cid:16) σ (cid:96)u h ( (cid:96),t − 1) ( x ) (cid:17) (cid:62) (cid:13)(cid:13)(cid:13)(cid:13)
F + 1 √ n (cid:13)(cid:13)(cid:13) δ ( (cid:96),t ) ( x ) · σ (cid:96)b (cid:13)(cid:13)(cid:13) F (cid:19)
+
T (cid:88) t
= (cid:18)n (cid:13)(cid:13)(cid:13)(cid:13) δ (1 ,t ) ( x ) (cid:16) σ 1 w h (1 ,t − 1) ( x ) (cid:17) (cid:62) (cid:13)(cid:13)(cid:13)(cid:13)
F + 1 √ nm (cid:13)(cid:13)(cid:13) δ (1 ,t − 1) ( x ) (cid:0) σ 1 u x t (cid:1) (cid:62) (cid:13)(cid:13)(cid:13) F + 1 √ n (cid:13)(cid:13)(cid:13) δ (1 ,t ) ( x ) · σ 1 b (cid:13)(cid:13)(cid:13) F (cid:19)
+ σ v
√ n (cid:107) h ( L,T ) ( x ) (cid:107) F ≤ (cid:18) L (cid:88) (cid:96) = T (cid:88) t = ( K 21 Cσ (cid:96)w + K 21 Cσ (cid:96)u
[Equation]
[Table]
[Equation]
The above proof can be generalized to the entire dataset by a straightforward application of the union bound.
This concludes the proof for Theorem 2.