SGORNN: C OMBINING S CALAR G ATES AND O RTHOGONAL C ONSTRAINTS IN R ECURRENT N ET - WORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recurrent Neural Network (RNN) models have been applied in different domains, producing high accuracies on time-dependent data.
However, RNNs have long suffered from exploding gradients during training, mainly due to their recurrent process.
In this context, we propose a variant of the scalar gated FastRNN architecture, called Scalar Gated Orthogonal Recurrent Neural Networks (SGORNN).
SGORNN utilizes orthogonal linear transformations at the recurrent step.
In our experiments, SGORNN forms its recurrent weights through a strategy inspired by Volume Preserving RNNs (VPRNN), though our architecture allows the use of any orthogonal constraint mechanism.
We present a simple constraint on the scalar gates of SGORNN, which is easily enforced at training time to provide a theoretical generalization ability for SGORNN similar to that of FastRNN.
Next, we provide bounds on the gradients of SGORNN, to show the impossibility of (exponentially) exploding gradients.
Our experimental results on the addition problem conﬁrm that our combination of orthogonal and scalar gated RNNs are able to outperform both predecessor models on long sequences using only a single RNN cell, and outperforms LSTM.
We further evaluate SGORNN on the HAR-2 classiﬁcation task, where it improves upon the accuracy of FastRNN, VPRNN, and LSTM using far fewer parameters than FastRNN or LSTM.
Finally, we evaluate SGORNN on the Penn Treebank word-level language modelling task, where it again outperforms its predecessor architectures and shows comparable performance to LSTM using far less parameters.
Overall, this architecture shows higher representation capacity than VPRNN, suffers from less overﬁtting than the other two models in our experiments, beneﬁts from a decrease in parameter count, and alleviates exploding gradients when compared with FastRNN on the addition problem.
I NTRODUCTION
Recurrent Neural Networks (RNN) are models for sequence processing that have found applications in many areas, such as statistical machine translation (Cho et al., 2014), credit score calculation (Babaev et al., 2019),
vehicle mobility prediction (Liu & Shoji, 2019), and music generation (Agarwala et al., 2017).
Despite their wide applicability, RNNs have long suffered from exploding gradients, mainly due to the repeated application of the network’s hidden weights matrix during gradient computation (Hochreiter & Schmidhuber, 1997).
Several strategies have been proposed to solve this problem, including models with orthogonal/unitary constraints on the hidden weights (Taylor-Melanson et al., 2021; Casado & Mart´ınez-Rubio, 2019; Zhang et al., 2018;
Vorontsov et al., 2017; Arjovsky et al., 2016; Jing et al., 2016).
Such orthogonal models usually have fewer parameters than standard RNN architectures, which reduces the representation capacity and leads to less accurate models.
Speciﬁcally, Taylor-Melanson et al.
(2021) noted that a stack of two RNN cells is required to accurately model the addition problem for RNNs using the orthogonal VPRNN architecture when sequences become very long, and Rusch & Mishra (2020) noted that the EXPRNN architecture (Casado & Mart´ınez-Rubio, 2019) fails to converge on the addition problem for very long sequences.
To combat the problem of reduced representation capacity, Jing et al.
(2019) proposed an LSTM-based architecture called the gated orthogonal recurrent unit (GORU) which was experimented with using the EURNN orthogonal parametrization (Jing et al., 2016).
While the GORU architecture is a very useful and novel orthogonal RNN construction, it adds many parameters when compared to typical orthogonal RNNs, and thus may not be suited for embedded or mobile applications.
Recently, Kusupati et al.
(2019) proposed the FastRNN architecture, which adds a residual connection to a standard recurrent network using a scalar gating mechanism.
FastRNN outperforms several well-known unitary RNN models on many classiﬁcation tasks.
These scalar gated RNN models have a generalization error gap which scales with the product of the length of the sequences being processed and the non-residual scalar gate, which is established through constraints introduced on the scalar gates of the network that are generally not enforced while training.
Because of this, the assumptions required to prove the generalization of FastRNN cannot be guaranteed to be satisﬁed in practice under the original formulation of the architecture.
Although the VPRNN architecture and many other RNNs alleviate exploding gradients by utilizing orthogonal constraints, the problem of increasing representation capacity for orthogonal RNNs without greatly increasing parameter counts still remains.
Moreover, even with the beneﬁt from the weighted residual connection, FastRNN can still suffer from exploding gradients when applied to long sequences as shown in Section 3.1, and Rusch & Mishra (2020) noted that FastRNN fails to converge when training on long sequences for the RNN addition problem.
In order to solve these problems, we propose the Scalar Gated Orthogonal Recurrent Neural Network (SGORNN), which is a variant of FastRNN that uses orthogonal linear transformations at the recurrent step.
The constraints on the scalar gates of FastRNN required for generalization are dependent on bounds of the Frobenius norm of the hidden weights matrix, and of the sequence length of the data being processed.
Both of these quantities may not be well-deﬁned in practice, making it difﬁcult to ensure generalization.
In contrast with FastRNN constraints, we present a constraint that relies only on the scalar gates of the SGORNN model, which is feasible to be employed in practice and results in a generalization gap with similar growth to that of FastRNN.
Our theoretical results also conﬁrm
that
SGORNN is immune to the (exponentially) exploding gradients so often seen when training RNN models.
In summary, SGORNN can ensure its theoretical generalization by applying the constraints we provide, which also alleviates exploding gradients.
Our addition problem experiment shows that
SGORNN converges for long sequences with a single RNN cell, outperforming both predecessor architectures despite having far fewer parameters than FastRNN.
Using real-world data, SGORNN shows superior generalization to its predecessor models and comparable performance to LSTM models with far more parameters.
The remaining sections of this paper are organized as follows.
In Section 2, we describe the proposed SGORNN architecture, followed by the theoretical formulation on generalization and exploding gradients.
Experiments are presented and discussed in Section 3.
In Section 4, we present the conclusions obtained from the experiments and formulation, as well as proposed future work.
Finally, Appendix A contains detailed information on the mathematical formulation used to provide the generalization bound for SGORNN, as well as the alleviation of exploding gradients.
SGORNN
The proposed Scalar Gated Orthogonal Recurrent Neural Network (SGORNN) uses orthogonal linear transformations at the recurrent step.
Given an input sequence { x t } Tt =1 ⊂ R n of length T > 0 , we deﬁne the hidden states h t ∈ R h of a SGORNN at time t as:
[Equation]
where Θ holds the recurrent parameters of the architecture, φ is a mapping from recurrent parameters into the orthogonal matrices,
W ∈ R h × n and U
= φ (Θ) ∈
R h × h are matrices, b ∈ R h is a bias vector, and σ is a nonlinear activation function, such as rectiﬁed linear unit (ReLU).
We use the notation U =
φ (Θ) for consistency with Kusupati et al.
during analysis.
As with FastRNN, the parameters α, β ∈ (0 , 1) are set by the sigmoid function.
For simplicity, we assume h 0 is a vector of zeros, though, in theory, it can be made to be a trainable value as well.
For the sake of experimentation, we choose φ (Θ) to be deﬁned as the hidden weights matrix of the
VPRNN architecture (Taylor-Melanson et al., 2021), a recently developed orthogonal RNN which is known to outperform several other unitary/orthogonal RNNs on classiﬁcation tasks.
It is worth mentioning that, although we use the VPRNN mapping in our experiments with SGORNN, an alternative orthogonal constraint such as that of EXPRNN (Casado & Mart´ınez-Rubio, 2019) (with φ = exp ) can also be applied to the proposed architecture.
Following from Taylor-Melanson et al.
(2021), the recurrent matrix is deﬁned as:
[Equation]
which is a case of SGORNN with dom ( φ )
=
R n/ 2 × k , such that R j performs pairwise 2D rotations of the input vector using parameters from the j th column of Θ , and Q j is a ﬁxed permutation matrix.
This formulation is inspired by a feed-forward predecessor (MacDonald et al., 2021).
It should be noted that, in the publicly available implementation of the VPNN layers from MacDonald et al.
(2021) written in Keras (Chollet et al., 2015) that we require to implement SGORNN 1 , a modiﬁcation to the permutation structure of U is present in the code.
This detail is ignored to simplify notation, but is justiﬁed in the supplementary material for the sake of proving Corollary 1.
The constraint we propose for the scalar gates of SGORNN to ensure generalization takes the following form:
[Equation]
At ﬁrst glance, this constraint may seem to limit the representation capacity of the model as FastRNN takes β = 1 − α > 1 − 2 α by assumption during theoretical analysis.
However, typically α (cid:28) 1 , in which case β + 2 α ≈ β + α , showing that an approximation to the standard formulation of the scalar gates remains.
This constraint is empirically justiﬁed in later sections.
Because our proposed constraints on α and β are independent of all other parameters (or bounds), they are very simply enforced at training time by applying Equation 3.
In practice, the parameter β is clipped after applying the sigmoid function, so that it lies in the half-open interval (0 , 1 − 2 α ] .
2. G ENERALIZATION B OUND
Initially, we state the constraint required for the generalization of FastRNN, to contrast with the constraints we require to guarantee the generalization of SGORNN and avoid exploding gradients.
It is assumed for FastRNN that (cid:107) U (cid:107) F ≤ R U for some R U > 0 , and the required constraint takes the form:
[Equation]
In the above equation, D is a real number derived from the parameters and activation function of the model.
From this, Kusupati et al.
(2019) established an O ( αT ) generalization bound for FastRNN which becomes independent of T since α = O (1 /T ) .
However, establishing this constraint in practice is difﬁcult for several reasons.
First, the constraint is dependent on the sequence length T of the model, which may be ill-deﬁned in real-world scenarios, and can lead to vanishing α for very long sequences.
Second, it relies on restricting the function space to those RNNs with (cid:107) U (cid:107) F ≤ R U , so that R U is well-deﬁned for use in Equation 4.
In addition, in many cases, choosing such an
R U before starting the training may decrease representation capacity drastically or, again, lead to vanishing α .
Finally, the constraint becomes undeﬁned for orthogonal matrices ( (cid:107) U (cid:107) 2 = 1 ).
These issues motivate the use of our constraint from Equation 3 for SGORNN, which depends only on the use of the scalar gates α and β , and can be enforced at training time through a simple clipping operation without any prior assumptions about the sequence length or other parameters of the RNN.
We deﬁne the class F T of SGORNN networks with ReLU activations as follows.
Given bounds
R W , R x > 0 , any ﬁxed α, β ∈ (0 , 1) with β ≤ 1 − 2 α , a mapping φ from parameters to orthogonal matrices, and an input space X of sequences { x t } Tt =1 where (cid:107) x t (cid:107) 2 ≤ R x , the SGORNN function class is deﬁned as:
[Equation]
It is implicit in the deﬁnition of the SGORNN function class that bias vectors are omitted or absorbed into the weights matrix, σ is the ReLU activation, and that the dimensions of h T − 1 , x T , and W are appropriate.
The Rademacher complexity of this function class F T leads to the following theorem, which describes the generalization ability of SGORNN models with ReLU activation functions and with a single output neuron using any 1-Lipschitz activation function.
This provides the equivalent of the FastRNN generalization gap result for our new proposed architecture:
Theorem 1 Let h T denote the ﬁnal hidden state of a SGORNN model from the function class F T for predetermined bounds R W , R x .
For any vector v ∈ R h with (cid:107) v (cid:107) 2 ≤ R v , and any 1-Lipschitz activation function σ y , denote f ∈ σ y ◦ v ◦ F T as a single-neuron output model such that f ( X ) =
σ y ( v (cid:62) h T ) .
Then, denoting L : R × R → [0 , B L ] as any bounded 1-Lipschitz loss function for some B L > 0 , we have with probability at least 1 − δ over the draw of an i.i.d.
sample S =
{ ( X ( i ) , y ( i ) ) } mi =1 from any input-output distribution, the following generalization gap holds:
[Equation]
where C = R x R W R v represents the boundedness of the model parameters and input data, as in the analysis of FastRNN (Kusupati et al., 2019), and the expectation is taken over the input-output distribution.
The presented theoretical result for SGORNN shows that the generalization of the model is independent of U and Θ , which were not used to compute the bounds on α and β and do not appear in the deﬁnition of C .
This is because the spectral norm for matrices is consistent with the Euclidean vector norm and (cid:107) U (cid:107) 2 = 1 for SGORNN.
Further, the constraints on the scalar gates from Equation 3 do not depend on the sequence length T of the model.
A proof of Theorem 1 is provided in
Appendix A.1.
2. E XPLODING G RADIENTS
Next, we analyze the gradients ∂L∂ W and ∂L∂ Θ of a loss function L , which is assumed to be 1-Lipschitz.
As W and U are the only parameter matrices interacting with the loss through temporal layers, we study the maximum entry norm (cid:107)·(cid:107) max of the loss function with respect to these parameters to show that SGORNN does not suffer from exploding gradients through time when applying a weaker constraint on α and β than the one from Equation 3.
This analysis leads to our second theorem:
Theorem 2 Let f ∈ σ y ◦ v ◦ F T be a SGORNN model with ReLU activation functions (as in
Theorem 1) with a trivialization mapping φ satisfying (cid:13)(cid:13)(cid:13) ∂φ (Θ) ∂
Θ ij (cid:13)(cid:13)(cid:13) max ≤ M φ for all indices i, j , and with β ≤ 1 − α .
Then, the maximum parameter gradients of a 1-Lipschitz loss L satisfy:
[Equation]
Where C and R W are deﬁned as in Theorem 1.
From such a theorem, we obtain the following corollary concerning our chosen implementation of
SGORNN:
Corollary
The VPRNN trivialization satisﬁes Theorem 2 with M φ = 4 √
h log 2 ( h ) , and so the gradients of the parameters Θ satisfy:
[Equation]
The theoretical bounds on the gradients above follow from the gradient computations provided in
FastRNN (Kusupati et al., 2019) combined with knowledge of the VPRNN architecture (TaylorMelanson et al., 2021).
The proof of Theorem 2 is provided in Appendix A.2, while the details of
Corollary 1 are in Appendix A.3.
It should be noted that the above bounds differ from the ﬁndings of Arjovsky et al.
(2016), who provided gradient bounds on general unitary RNN models independent of T .
This is because the gradient computation of FastRNN (and consequently SGORNN) is greatly inﬂuenced by the residual connection through α and β .
Regardless of this complication, we have shown that SGORNN does not suffer from the exponentially exploding gradients often seen when training RNNs.
E XPERIMENTS 3. A DDITION P ROBLEM
As a proof of concept, the ﬁrst experiment considered was the addition problem for RNNs, which was ﬁrst proposed by Hochreiter & Schmidhuber (1997).
This problem takes the form of a regression task, typically on the output vector h T of a recurrent network.
The input sequence x t is a 2D sequence x t = ( u t c t ) (cid:62) , with u t being uniformly distributed in [0 , 1] (i.i.d.) and c t representing a binary sequence.
This binary sequence is mostly ﬁlled with 0 s, having only two elements at positions i and j set as 1 , in which i is chosen uniformly in [1 , T/ 2] and j chosen uniformly in ( T/ 2 , T ] .
Intuitively, the addition problem gives the sum of two random target positions in the sequence, that the RNN model must recognize and maintain (remember) over time.
The addition problem has been used as a benchmark for many RNN models which aim to process long sequences, as the randomly generated data introduces long-term dependencies (Li et al., 2018;
Taylor-Melanson et al., 2021; Hochreiter & Schmidhuber, 1997; Arjovsky et al., 2016).
A model applied on the addition problem is considered converged if it can consistently and considerably beat the baseline (MSE) error of approximately 0 .
167 on unseen data (Taylor-Melanson et al., 2021;
Li et al., 2018), which is the error when the mean of the target ( y = 1 ) is always predicted.
In our experiments, all data is randomly generated as performed in the recent literature (Taylor-Melanson et al., 2021; Li et al., 2018).
The regression target y over the sequence is the sum of the two random numbers as indicated by the indices i, j , deﬁned mathematically as:
[Equation]
We compared the proposed SGORNN with VPRNN, FastRNN, and LSTM using the same model conﬁguration and batch size for all architectures.
In this experiment, all models had a hidden state dimension of 128 , and the SGORNN and VPRNN used k = 14 rotational sublayers, which is the maximum number allowed based on the constraint from VPRNN architecture (Taylor-Melanson et al., 2021).
Every model consisted of just one RNN layer with ReLU as the activation function.
The constraint from Equation 3 was imposed on the SGORNN model so that the model satisﬁes Theorem 1.
The RMSprop optimizer (Tieleman & Hinton, 2012) was used, in which the learning rate was decayed linearly to zero during training.
Similarly to experiments conducted by Taylor-Melanson et al.
(2021), models were trained with a batch size of 64 for a maximum of 20 , 000 training steps, and 10 batches of unseen data were used to record validation MSE after every 100 training steps.
We note that all models trained by Li et al.
(2018) that converged on the addition problem did so within 20 , 000 training steps, motivating our choice of cutoff for measuring convergence.
The goal of this experiment was to show which of the models tested are able to converge on different sequence lengths.
For this reason, initial learning rates of each model were ﬁrst set to 10 − 2 , and were decreased by a factor of 10 each time a model could not converge.
For example, if a model could not converge with the initial learning rate of 10 − 2 it was retrained with an initial learning rate of 10 − 3 .
This was allowed to be done twice for each model, so that all initial learning rates came from the set { 10 − 2 , 10 − 3 , 10 − 4 } .
In this way, we performed a basic search for the maximum learning rate to allow the convergence of the models.
However, due to issues such as exploding gradients, some models could not converge for this task.
In the case of partial convergence (e.g. VPRNN with
T = 1000 , see Figure 2), a learning rate decrease was performed to ensure better convergence could not be obtained.
For such a scenario, the result for the best learning rate was reported.
Figures 1, 2, and 3 present the validation error obtained for each model using the sequence lengths 500 , 1000 , and 5000 respectively.
For the shortest sequence length, T = 500 , all three models were able to converge using the largest initial learning rate of 10 − 2 , though FastRNN and SGORNN provided the most efﬁcient convergence.
In the case of sequences of length T = 1000 , VPRNN failed to converge with learning rates of 10 − 2 and 10 − 4 , though it achieved partial convergence with a learning rate of 10 − 3 .
Also with T = 1000 , SGORNN converged efﬁciently with the largest learning rate of 10 − 2 .
FastRNN suffered from exploding gradients and divergent loss using all learning rates, even with the smallest learning rate of 10 − 4 .
FastRNN also achieved partial convergence for T = 1000 as shown in Figure 2, but was unable to reach an efﬁcient solution before exploding gradients occurred.
Lastly, for the largest sequence length of T = 5000 , we observed that FastRNN suffered from exploding gradients and consequently a divergent MSE loss for all initial learning rates.
VPRNN also failed to converge for T = 5000 regardless of the initial learning rate chosen, though it did not suffer from exploding gradients.
In contrast to this result, SGORNN still converged efﬁciently using the largest initial learning rate of 10 − 2 and did not suffer from exploding gradients.
The LSTM model failed for all sequence lengths greater than T = 500 regardless of the chosen learning rate.
[Table]
3. HAR-2 C LASSIFICATION
The HAR-2 classiﬁcation task is a binary classiﬁcation task derived from a human movement data set collected using smartphones (Anguita et al., 2013).
The data set consists of 9 -dimensional sequences of length T = 128 , having 7 , 352 training instances and 2 , 947 testing instances.
We used this dataset to train and evaluate SGORNN, FastRNN, VPRNN, and LSTM models, conducting a performance comparison between them.
All models were single-layer RNNs with 128 hidden units, except for the LSTM which was chosen to have 64 hidden units to roughly match the number of parameters of the FastRNN, similar to how hidden state sizes were chosen for a language modelling task in
GORU (Jing et al., 2019).
The SGORNN and VPRNN models had k = 14 rotational sublayers, which is the maximum allowed following from the constraint presented by Taylor-Melanson et al.
We used an RMSprop optimizer (Tieleman & Hinton, 2012) with an initial learning rate of 10 − 3 .
Models were trained for 300 epochs using a batch size of 100 , and the learning rate was decreased by a factor of 10 after 200 epochs.
This strategy is similar to the one employed for FastRNN (Kusupati et al., 2019) with the main difference being our lack of hyperparameter tuning.
We trained each model 10 times using different random seeds.
The constraint from Equationwas imposed on the SGORNN model to ensure good generalization based on Theorem 1.
This experiment shows that our proposed constraint from Equation 3 can be applied in practice using a real-world data set.
As shown in Table 2, SGORNN achieved the highest mean test accuracy among the three models trained.
Although this accuracy is not drastically above that of FastRNN or LSTM, across thetraining runs, the standard deviation of the test accuracies measured was the lowest for SGORNN.
This suggests that SGORNN had both greater performance and more stable training than the other models, while requiring far fewer parameters than FastRNN or LSTM.
We use the difference between the training and test performance (for both accuracy and loss) to estimate the degree of overﬁtting of the models trained on this task.
Plots of these estimates of overﬁtting are illustrated in Figure 4.
Interestingly, VPRNN has the largest gap between training and testing accuracy, although this gap does not signiﬁcantly differ from that of FastRNN or LSTM.
[Table]
Further, the accuracy gap for both FastRNN, VPRNN, and LSTM seems to increase as training goes on, while the gap for SGORNN seems to decrease and eventually converge to a stationary value between 3 % and 4 % (Figure 4a).
Even more interesting is the degree of overﬁtting of FastRNN with respect to the entropy loss, where the decrease in learning rate after 200 epochs caused the gap to grow much faster.
VPRNN, FastRNN, and LSTM have an entropy gap growing with the number of epochs, while the SGORNN model loss gap remains nearly constant and has a very small variance (Figure 4b).
This behavior suggests that, as training goes on, the degree of overﬁtting of SGORNN decreases or remains constant while it increases for all other models.
[Figure]
3. P ENN T REEBANK L ANGUAGE M ODELLING
The Penn Treebank (PTB) data set is an annotated corpus for natural language processing tasks (Marcus et al., 1993).
The word-level version of the data set consists of approximately 929 , 000 training tokens, 74 , 000 validation tokens, and 83 , 000 testing tokens.
The most common 10 , 000 tokens are used, with the rest of the tokens replaced by <unk> tokens.
Newline characters are replaced with <eos> tokens.
We use this data set to evaluate our models on a language modelling task, where the output of the RNN at time t is a probability distribution over the token to be observed at time t +calculated using a softmax activation function on a fully-connected output layer.
We take our sequence length and model architecture from FastRNN (Kusupati et al., 2019), processing sequences of length 300 and ﬁxing the dimension of the trainable embeddings of each model to be the same as that of the hidden states of the RNN being trained.
Our models were all single-cell
RNNs with 256 hidden units.
Because the classiﬁcation layer of these language models accounts for such a large proportion of the overall parameters, the LSTM model was allowed to remain athidden units (unlike in the previous HAR-2 task).
The SGORNN and VPRNN models had k =rotational sublayers.
The constraint from Equation 3 was enforced on SGORNN to ensure generalization according to Theorem 1.
We used the RMSprop optimizer (Tieleman & Hinton, 2012), with the same number of epochs and learning rate decay scheme as with the HAR-2 experiment (Section 3.2).
As an attempt to prevent overﬁtting in all models, we applied dropout of 0 .
3 to the embeddings given as input to the RNN cells.
The metric used to evaluate models is the test perplexity, which measures the level of uncertainty of the language models (lower is better).
It is seen in
Table 3 that SGORNN outperforms both of its predecessor models, and shows comparable performance to LSTM despite having only a small fraction of the number of parameters within its RNN cell. [Table]
C ONCLUSIONS
This paper has presented a new RNN architecture which combines orthogonal constraints with a scalar gating mechanism.
This new RNN, which we call SGORNN, was implemented for the purpose of experimentation using an orthogonal transformation scheme from recent literature called
VPRNN.
However, as only Corollary 1 relies directly on the structure of the VPRNN mapping, our main theoretical results from Theorems 1 and 2 can be applied with different mapping strategies, such as the EXPRNN mapping (Casado & Mart ´ ınez-Rubio, 2019).
Our theoretical analysis presented both bounds on the gradients of SGORNN and a generalization gap bound for Lipschitz loss functions.
The gradient bound for SGORNN shows that gradients do not explode exponentially through time, but instead are O (( αT ) 2 ) in the worst case when α ≤ 1 − β .
This constraint to control gradients is weaker than the one required for generalization of our architecture, in that enforcing the generalization constraint also enforces the gradient bound constraint.
In theory, through further controlling bounds on α , we can fully control the upper gradient bound presented.
The constraint on α (Equation 3) depends only on the other scalar gate β , rather than the α = O (1 /T ) constraint required for generalization of a FastRNN, which depends on bounds constructed for other trainable parameters.
Because of this dependence, our constraint for generalization was easily enforced through a clipping operation for the three tasks in our experiments.
In our experiments, we ﬁrst trained models on the addition problem, where SGORNN was able to converge on sequences of length T = 5000 using less than 1 , 500 parameters, while FastRNN suffered from exploding gradients and both VPRNN and LSTM failed to converge outright.
This is the smallest number of parameters that was found in the literature for RNNs able to process such long sequences on the addition problem.
Further, it shows that SGORNN may require just one RNN cell to perform accurately on tasks where other architectures with a reduced number of parameters such as IndRNN (Li et al., 2018) and VPRNN (Taylor-Melanson et al., 2021) have been noted to require two stacked cells.
On the HAR-2 classiﬁcation task, the SGORNN model had both the highest accuracy and suffered from the least amount of overﬁtting of all models considered.
On the PTB word-level language modelling task, SGORNN outperformed both VPRNN and FastRNN in terms of test perplexity, and shows comparable performance to our LSTM.
It is worth noting that our test perplexity for FastRNN is higher than that reported by Kusupati et al.
(2019), which may be attributed to differences in hyperparameters.
Based on the experiments, we conclude that the SGORNN architecture brings enforceable generalization and well-controlled gradients to the FastRNN architecture, and brings increased representation capacity to orthogonal RNN architectures such as VPRNN, reinforcing our theoretical results.
Future work on SGORNN will include further investigation of generalization ability, as well as investigation of alternative constraints on the scalar gates.
Additional architectures such as SGORNN models with standard gating mechanisms may also be investigated.
Finally, convergence analysis of SGORNN and the investigation of mappings to orthogonal matrices other than that of VPRNN will also be the topic of future work.
A P ROOFS OF S TATED T HEOREMS AND D ETAILS OF C OROLLARIES
In the following proofs we follow notation similar to (Kusupati et al., 2019).
This is different from that of (Taylor-Melanson et al., 2021) who denote the recurrent weights matrix by V rather than
U .
We choose to denote the empirical Rademacher complexity of a function class F by R m ( F ) , to indicate that the average is taken over a random sample of size m .
This differs slightly from both (Kusupati et al., 2019) and (Bousquet et al., 2003) who use n instead of m to denote the size of the random sample since we use n to describe the input data dimension in Section 2.
A. P ROOF OF T HEOREM
We ﬁrst begin with a description of the one-sided Rademacher complexity bound framework used.
For any probability distribution D over input-output pairs z
=
( X
, y ) ∈ Z , any bound B L > 0 , and any loss function L : Z × ˆ Y → [0 , B L ] taking in a training instance and a prediction ˆ
y ∈
ˆ Y , deﬁne a class of functions over any class of predictors F taking values in ˆ Y as:
[Equation]
so that if (cid:96)
f ∈ L ◦F then (cid:96) f : Z → [0 , B L ] is a mapping from input-output pairs to loss values.
The empirical Rademacher complexity of this class of functions is deﬁned as in (Bousquet et al., 2003) on a sample S
= { z (1) , . . .
, z ( m ) } drawn i.i.d.
from D as:
[Equation]
The main result we require to prove the generalization of SGORNN is the one-sided Rademacher complexity bound from Bartlett & Mendelson (2002), which is also used by FastRNN (Kusupati et al., 2019).
It states that, for all δ > 0 , with probability at least 1 − δ over the draw of S , for any (cid:96)
f ∈ L ◦ F and distribution D over Z :
[Equation]
Theorem 1 Let h T denote the ﬁnal hidden state of a SGORNN model from the function class F T for predetermined bounds R W , R x .
For any vector v ∈ R h with (cid:107) v (cid:107) 2 ≤ R v , and any 1-Lipschitz activation function σ y , denote f ∈ σ y ◦ v ◦ F T as a single-neuron output model such that f ( X ) =
σ y ( v (cid:62) h T ) .
Then, denoting L : R × R → [0 , B L ] as any bounded 1-Lipschitz loss function for some B L > 0 , we have with probability at least 1 − δ over the draw of an i.i.d.
sample S =
{ ( X ( i ) , y ( i ) ) } mi =1 from any input-output distribution, the following generalization gap holds:
[Equation]
where C = R x R W R v represents the boundedness of the model parameters and input data, as in the analysis of FastRNN (Kusupati et al., 2019), and the expectation is taken over the input-output distribution.
It just remains to establish a bound on R m ( L ◦ F ) .
As before, denote f
( X ( i ) )
=
σ y ( v (cid:62)
h
( i ) T ) as the output of the model on a given input sequence
X ( i ) =
( x ( i ) 1 , . . .
, x ( i ) T ) , where h ( i ) T denotes the ﬁnal hidden state of the SGORNN on the given input.
Denote the class of SGORNN networks (outputting h T ) as F T , so that F = σ y ◦ v ◦ F T is the overall class of hypothesis functions.
Because both L and σ y are assumed to be 1-Lipschitz, we have by the Ledoux-Talagrand contraction (the needed version of which has a proof in (Mohri & Medina, 2014))
that:
[Equation]
By deﬁnition of the empirical Rademacher complexity, we then have that:
[Equation]
Next, recall that (cid:107) v (cid:107) 2 ≤ R v
and (cid:107) W (cid:107) F ≤ R W .
There is no explicit restriction placed on Θ as in the deﬁnition of F T .
Note that:
[Equation]
Now, we follow a similar strategy employed for FastRNN (Kusupati et al., 2019).
Denoting
R m ( F T ) as the above expected value, we have modiﬁed the notation of an inequality for R m ( F T ) , provided by Kusupati et al.
, to capture the supremum over recurrent parameters Θ :
[Equation]
Equation 20 relies on an analog of the Ledoux-Talagrand contraction to remove dependence on the activation function σ .
The ﬁrst expected value in this equation is bounded by R W R x √ m using the same argument as Kusupati et al.
Because U is orthogonal, the second expected value in
Equation 20 is:
[Equation]
So that
[Equation]
by substituting the bounds on the expected values [Equation] Equation 20 becomes:
[Equation]
Now, using the fact that β ≤ 1 − 2 α , the coefﬁcient of R m ( F T − 1 ) in the above recurrence is at most 1 .
Assuming h ( i )
= 0 for all i , we get that R m ( F 0 )
=
and consequently:
[Equation]
Setting C
= R v R W R x , combining Equation 19 and Equation 22 gives a bound on the Rademacher complexity of L ◦ F of:
[Equation]
and ﬁnally, the desired bound holding with probability at least 1 − δ comes from combining Equations 13 and 23:
[Equation]
which is equivalent to the original statement of the theorem, as (cid:96)
f
( z ) is simply shorthand for
L ( y, f ( X )) .
(cid:4)
A. P ROOF OF T HEOREM
The proof relies on the two following formulas, taken from the formulation presented by Kusupati et al.
[Equation]
where D k = diag ( σ (cid:48)
( Wx k + Uh k − 1 )) is the Jacobian matrix of the pointwise activation function σ .
Note that for ReLU, D k is a diagonal matrix with a binary vector on the diagonal.
Theorem 2 Let f ∈ σ y ◦ v ◦ F T be a SGORNN model with ReLU activation functions (as in
Theorem 1) with a trivialization mapping φ satisfying (cid:13)(cid:13)(cid:13) ∂φ (Θ) ∂
Θ ij (cid:13)(cid:13)(cid:13) max ≤ M φ for all indices i, j , and with β ≤ 1 − α .
Then, the maximum parameter gradients of a 1-Lipschitz loss L satisfy:
[Equation]
Where C and R W are deﬁned as in Theorem 1.
We begin by establishing the ﬁrst bound involving Θ .
Note that by the chain rule:
[Equation]
Note also that we simply denoted U
= φ (Θ) , so:
[Equation]
Thus, since (cid:12)(cid:12)(cid:12)
[Equation]
where we use the fact that (cid:107) X (cid:107) 1 , 1 ≤ √ h (cid:107) X (cid:107) F and (cid:107) X (cid:107) F ≤ √ h (cid:107) X
(cid:107) 2 for any h × h matrix X .
Next, we use the subadditivity and submultiplicativity of the spectral norm applied to Eq. 25:
[Equation]
+ β (cid:107) I (cid:107) 2 ) (cid:33) (cid:13)(cid:13) ( ∇ h T L )
h (cid:62) t − 1 (cid:13)(cid:13) 2 .
Because ReLU is 1-Lipschitz, (cid:107) D k (cid:107) 2 ≤ 1 for any k .
Further, (cid:13)(cid:13) U (cid:62) (cid:13)(cid:13)
= (cid:107) I (cid:107) 2 = 1 .
Thus, the product in the formula has an upper bound of ( α + β ) T − t ≤ 1 .
We also have that (cid:13)(cid:13) ( ∇ h T L ) h (cid:62) t − 1 (cid:13)(cid:13) 2 ≤ (cid:107)∇ h T L (cid:107) 2 (cid:107) h t − 1 (cid:107) 2 .
Thus:
[Equation]
∂L ∂ U (cid:13)(cid:13)(cid:13)(cid:13) 2 ≤ α (cid:107)∇ h T L (cid:107) T (cid:88) t = (cid:107) h t − 1 (cid:107) 2 . It is then left to bound (cid:107)∇ h T L (cid:107) 2 and (cid:107) h t − 1 (cid:107) 2 .
First, for the hidden state: {+(cid:107) h t − 1 (cid:107) 2 ≤ (cid:107) ασ ( Wx t − 1 + Uh t − 2 ) + β h t − 2 (cid:107)≤ α (cid:107) σ ( Wx t − 1 + Uh t − 2 ) (cid:107) 
 + β (cid:107) h t − 2 (cid:107)≤ α (cid:107) Wx t − 1 + Uh t − 2 (cid:107) 2 + β (cid:107) h t − 2 (cid:107)≤ α ( R W+}
[Equation]
Solving the above recurrence gives (cid:107) h t − 1 (cid:107) 2 ≤ αTR W R x .
Next for the gradient, assuming a 1Lipschitz loss, we get that:
[Equation]
So that (cid:107)∇ h T L (cid:107) 2 ≤ R v .
Plugging these back in gives:
[Equation]
and consequently the gradient bound:
[Equation] [Equation]
as desired (the statement regarding the max norm follows since we chose arbitrary i, j ).
Moving on to the bound on the gradient for W , we get using the same strategy that: [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
as desired.
(cid:4)
A. D ETAILS OF C OROLLARY
Taylor-Melanson et al.
(2021) showed that using the VPRNN trivialization, for any Θ and ∆Θ :
[Equation]
(34) where k = 2 (cid:100) log 2 h (cid:101) .
Note that k ≤ 4 log 2 h for h ≥ 2 which can be safely assumed.
Now, consider a perturbation ∆Θ with zero entries everywhere but the i, j th entry, so that:
[Equation]
[Equation]
The numerator has the upper bound:
[Equation]
while the denominator has the lower bound (since ∆Θ has only one nonzero entry):
[Equation]
Therefore, assuming the gradient of φ is deﬁned, and the limit exists, it must satisfy:
[Equation]
by the derivations presented for VPRNN (Taylor-Melanson et al., 2021).
This completes the derivation of the M φ bound for the VPRNN trivialization mapping.
A. J USTIFICATION OF C ODE I MPLEMENTATION
The public code implementation of the VPRNN layer we required for our experiments adds some additional permutations to the matrix decomposition of U = φ (Θ) .
Regardless of how many it adds, we can deﬁne the mapping ˜ φ of the implementation as ˜ φ = ˜Q 1 φ ˜Q 2 for permutation matrices ˜Q 1 , ˜Q 2 , where φ is a typical VPRNN mapping using suitable permutation matrices.
This deﬁnition is due to the fact that the product of two permutation matrices is also a permutation matrix.
Using the spectral norm (cid:107)·(cid:107) 2 , we get the following inequality since ˜Q 1 , ˜Q 2 are orthogonal: [Equation] ≤ [Equation]
∆Θ) ˜Q
[Equation]
−
[Equation]
(Θ) ˜Q
[Equation]
≤
[Equation]
∆Θ)
[Equation]
[Equation] ≤ [Equation] So, the result required for Corollary 1 holds for the implementation provided and the extra permutations can be ignored.