T RAINING RNN S AS F AST AS CNN S
Anonymous authors
Paper under double-blind review
A BSTRACT
Common recurrent neural network architectures scale poorly due to the intrinsic difﬁculty in parallelizing their state computations.
In this work, we propose the
Simple Recurrent Unit (SRU) architecture, a recurrent unit that simpliﬁes the computation and exposes more parallelism.
In SRU, the majority of computation for each step is independent of the recurrence and can be easily parallelized.
SRU is as fast as a convolutional layer and 5-10x faster than an optimized LSTM implementation.
We study SRUs on a wide range of applications, including classiﬁcation, question answering, language modeling, translation and speech recognition.
Our experiments demonstrate the effectiveness of SRU and the tradeoff it enables between speed and performance.
We open source our implementation in PyTorch and CNTK
I NTRODUCTION
Recurrent neural networks (RNN) are at the core of state-of-the-art approaches for a large number of natural language tasks, including machine translation (Cho et al., 2014;
Bahdanau et al., 2015; Jean et al., 2015; Luong et al., 2015), language modeling (Zaremba et al., 2014; Gal & Ghahramani, 2016;
Zoph & Le, 2016), opinion mining (Irsoy & Cardie, 2014), situated language understanding (Mei et al., 2016; Misra et al., 2017), and question answering (Seo et al., 2016a; Chen et al., 2017).
Key to many of these advancements are architectures of increasing capacity.
However, these networks are difﬁcult to scale.
During learning, the sequential dependencies that are central to recurrent architectures limit parallelization potential.
This results in slow development and makes rigorous parameter tuning intractable.
Similar problems occur during deployment when slow inference creates challenges for real-time systems at scale.
In this paper, we describe the Simple Recurrent Unit (SRU), a recurrent architecture that balances serial and parallelized computation.
We evaluate SRU and show the speed gains it provides generalize across a set of core tasks, while maintaining and even improving overall performance over common architectures.
Recurrent networks process sequences of symbols (e.g., words in a sentence) one symbol at a time.
In commonly used architectures, including Long Short-term Memory (LSTM; Hochreiter &
Schmidhuber, 1997) and Gated Recurrent Units (GRU; Cho et al., 2014), the computation in each step depends on completing the previous step.
As a result, in contrast to operations such as convolution and attention, recurrent computations are less amenable to parallelization.
We propose to do the majority of the computation for each step without depending on completing previous computations, which allows for to easily parallelize it.
The result of this computation are then combined via a fast recurrent structure.
Figure 1 illustrates the difference between the approaches.
While even a naive implementation of our approach leads to improvements in performance, one of its key advantage is enabling optimization particularly ﬁtting to existing hardware architectures.
Removing the dependencies between time steps for the most expensive operations allows to parallelize across different dimensions and time steps.
We also perform a CUDA-level optimization by compiling element-wise operations of the computations into a single kernel function call.
Figure 2 compares our architecture’s runtimes to common architectures.
We experiment with a diverse set of core problems to evaluate our architecture, including text classiﬁcation, question answering, language modeling, machine translation, and speech recognition.
Our approach is competitive and even outperforms common recurrent and convolutional architectures, while delivering signiﬁcant speedups.
We also study the relation between speed and performance, and show SRU provides ﬁne-grained control of the tradeoff between the two.
[Figure] [Figure]
M ETHOD 2. S IMPLE R ECURRENT U NITS
Most recurrent architectures, including LSTM and GRU, use gating to control the information ﬂow to alleviate vanishing and exploding gradient problems.
We deﬁne a gate to be composed of a single-layer feed-forward network with a sigmoid activation.
The gate output is used in a point-wise multiplication operation to combine two inputs, for example the current and previous time stamps.
The computation of the feed-forward network, especially the matrix multiplication, is the most expensive operation in this process, while the point-wise multiplication is relatively lightweight.
The key design decision in SRU is making the gate computation dependent only on the current input of the recurrence.
This leaves only the point-wise multiplication computation as dependent on previous steps.
The matrix multiplications involved in the feed-forward network can then be easily parallelized.
The basic form of SRU includes a single forget gate .
Given an input x t at time t , we compute a linear transformation ˜ x t (Lei et al., 2017; Lee et al., 2017) and the forget gate f t :
[Equation]
This computation depends on x t only, which enables computing it in parallel across all time steps.
The forget gate is used to modulate the internal state c t , which is used to compute the output state h t :
[Equation]
where g ( · ) is an activation function used to produce the output state h t .
The complete architecture also includes skip connections, which have been shown to improve training of deep networks with a large number of layers
(He et al., 2016; Srivastava et al., 2015; Wu et al., 2016a).
We use highway connections (Srivastava et al., 2015), and add a reset gate r t computed similar to the forget gate f t .
The reset gate is used to compute the output state h t as a combination of the internal state g ( c t ) and the input x t .
The complete architecture is:
[Equation]
2. R ELATION TO C OMMON A RCHITECTURES
Existing RNN architectures use the previous output state h t − 1 in the recurrence computation.
For example, in LSTM, the forget gate vector is computed by f t = σ ( W f x t + R f h t − 1 + b f ) .
Including
Rh t − 1 breaks independence and parallelization: each dimension of the hidden state h t depends on h t − 1 , and the computation of h t has to wait until h t − 1 is fully computed.
Similar design choices are present in GRU and other RNN variants, where h t − 1 is used throughout the computation.
We propose to completely drop the connection between the gating computations of step t and the states of step t − 1 , h t − 1 and c t − 1 .
Given a sequence of input vectors { x 1 , · · · , x n } , { ˜ x t , f t , r t } for different t = 1 · · · n are independent and can be computed in parallel.
The computation bottleneck of our architecture is the three matrix multiplications in Equations 1-3.
After computing ˜ x t , f t and r t ,
Equations 4 and 5, where all operations are element-wise, are fast to compute.
2. CUDA- LEVEL O PTIMIZATION
Optimizing SRU is similar to how LSTM is optimized in cuDNN LSTM (Appleyard et al., 2016).
The
SRU formulation permits two optimizations.
First, matrix multiplications across all time steps can be batched, which signiﬁcantly improves the computation intensity and GPU utilization.
Grouping the matrix multiplications in Equations 1-3 into a single batch is formulated as
[Equation]
where n is the sequence length, U ∈ R n × 3 d is the resulting matrix, and d is the hidden state size.
When the input is a mini-batch of k sequences, U would be a tensor of size ( n, k, 3 d ) .
Second, all element-wise operations of the sequence can be fused into one kernel function and parallelized across the dimensionality of the hidden state d .
Without the fusion, operations such as addition + and sigmoid activation σ () would each invoke a separate function call, and incur additional kernel launching latency and data moving costs.
Algorithm 1 shows the pseudocode of the fused kernel function.
The implementation of a bidirectional SRU is similar: the matrix multiplications of both directions are batched, and a fused kernel is created to handle and parallelize both directions.
R ELATED W ORK
Improving on common architectures for sequence processing has recently received signiﬁcant attention (Greff et al., 2015; Balduzzi & Ghifary, 2016; Miao et al., 2016; Zoph & Le, 2016; Lee et al., 2017; Vaswani et al., 2017).
Our approach is closely related to recent work on recurrent convolutions (RCNN; Lei et al., 2015; 2016), kernel networks (KNN; Lei et al., 2017), and QuasiRNN (Bradbury et al., 2017).
Both RCNN and Quasi-RNN incorporate word-level convolutions into recurrent unit with sequential gated pooling.
KNN generalizes RCNN and provides a theoretical view by linking the model class to sequence kernels.
SRU can be viewed as a simpliﬁed version, or a special case, of RCNN, KNN, and Quasi-RNN, where the window size is set to 1 and highway connections Srivastava et al.
(2015) are added to facilitate increased network depth.
We discuss the relation of SRU to Quasi-RNN in more detail and evaluate the effect of the differences in Appendix A.
Various strategies have been proposed to scale network training (Goyal et al., 2017) or speciﬁcally to speed up recurrent networks (Diamos et al., 2016; Kuchaiev & Ginsburg, 2017; Shazeer et al., 2017).
Our CUDA-level optimization for SRU is inspired by cuDNN LSTM (Appleyard et al., 2016).
While cuDNN LSTM requires six optimization steps, SRU only requires two optimizations to produce
Initialize h [ · , · , · ] and c [ · , · , · ] as two n × k × d tensors.
for i = 1 , · · · , k ; j = 1 , · · · , d do //
Parallelize over i and j c (cid:48)
=
c
[ i, j ] for l = 1 , · · · , n do f = σ ( U [ l, i, j + d ] + b f [ j ] ) //
Forget gate r = σ ( U [ l, i, j + d × 2] + b r [ j ] ) //
Reset gate c = f × c + (1 − f ) × U [ l, i, j ] //
Current internal state h
= f × g ( c ) + (1 − r ) ×
x
[ l, i, j ] // Current output state c [ l, i, j ] c h [ l, i j h
] signiﬁcant speed-up.
The convolution-based Quasi-RNN architecture (Bradbury et al., 2017) uses similar CUDA-level optimizations such as conv2d operation or batched matrix multiplications.
The topic of improving learning times was also studied.
For example, Goyal et al.
(2017) addressed stability issues of distributed training with large mini-batches to improve training time.
Our approach can be combined with such training procedures.
The design of simple recurrent architectures, such as SRU and other related architectures, raises questions about representational power.
Balduzzi & Ghifary (2016) applies type-preserving transformations to the discuss the capacity of various simpliﬁed RNN architectures.
Recent work has demonstrated the connection between neural networks and kernels (Anselmi et al., 2015; Daniely et al., 2016; Zhang et al., 2016).
In particular, Lei et al.
(2017) shows that a broad model class, including SRU and word-level CNN, can be seen as embedding sequence similarity functions, such as string kernels (Lodhi et al., 2002), into a hidden space.
Layer stacking can then be interpreted as using higher-order sequence similarities, which introduces more non-linearity and representational power.
We empirically show SRU can achieve compelling results by stacking multiple layers.
E XPERIMENTS
We evaluate SRU with text classiﬁcation, question answering, language modeling, machine translation, and speech recognition tasks.
This set of tasks provides broad coverage of application and computation challenges.
Training time on these benchmarks ranges from minutes (classiﬁcation) to days (speech).
Unless noted otherwise, timing experiments are performed on PyTorch and a desktop machine with a single Nvidia GeForce GTX
GPU, Intel Core i7-7700K Processor, CUDA 8 and cuDNN 6021.
We use variational dropout (Gal & Ghahramani, 2016) in addition to the standard dropout for RNN regularization.
We set g ( · ) =
tanh for all our experiments, unless speciﬁed otherwise.
The main question we study is the performance-speed trade-off SRU provides in comparison to other recurrent architectures.
We stack multiple layers of SRU to directly substitute other recurrent or convolutional modules.
We minimize hyper-parameter tuning and architecture engineering for a fair comparison.
Such efforts have a non-trivial impact on the results, which are beyond the scope of our experiments.
As much as possible, the model conﬁgurations are identical to prior work.
4. C LASSIFICATION
Dataset We use six classiﬁcation tasks from Kim (2014): 1 movie review sentiment (MR; Pang & Lee, 2005), subjectivity (SUBJ; Pang & Lee, 2004), customer reviews polarity (CR; Hu & Liu, 2004), TREC question type (TREC; Li & Roth, 2002), MPQA opinion polarity (MPQA; Wiebe et al.,
[Table]
[Figure]
, and the Stanford sentiment treebank (SST; Socher et al., 2013)
Following (Kim, 2014) we use word2vec embeddings trained on 100 billion Google News tokens.
Word embeddings normalized to unit vectors are ﬁxed during
Setup We train RNN encoders and use the last output state to predict the class label for a given sentence.
We use a two-layer RNN encoder with 128 hidden dimensions.
For SST, which provides more data, we use a four-layer RNN.
We also compare to the CNN model of Kim (2014), with the same ﬁlter windows of 3 , 4 , and 5 as the original work.
We use Adam (Kingma & Ba, 2014) with default 0 .
001 learning rate and 0 weight decay.
We train for 100 epochs, and perform 10 -fold cross validation when no standard split is speciﬁed.
The result on SST is averaged over ﬁve independent trials.
We tune dropout probability among { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 } and report the best results.
Results Table 1 presents test results on the six benchmarks.
Our model consistently outperforms the other models across the datasets.
Figure 2 shows validation performance relative to training time for SRU, cuDNN LSTM , and the CNN model.
Our SRU implementation is signiﬁcantly faster than cuDNN LSTM .
For example, on the movie review task (MR), our model completes 100 training epochs within 40 seconds, while cuDNN LSTM takes more than 450 seconds.
[Table]
State-of-the-art Results
[Table]
et al
[Table]
2016b
[Table] et al [Table] 4. Q UESTION A NSWERING
Dataset We use the Stanford Question Answering Dataset (SQuAD; Rajpurkar et al., 2016).
SQuAD is one of the largest machine comprehension datasets, and includes over 100K questionanswer pairs extracted from Wikipedia articles.
We use the standard train and development sets.
Setup We experiment with the Document Reader model (Chen et al., 2017), and compare variants that use LSTM, as in the original setup, and SRU.
We use the open source re-implementation.
3 Due to minor differences, this version performs 1% worse compared to the reported results when using the same training options.
Following the author suggestions, we use a learning rate of 0 .
001 instead of 0 .
002 , the Adamax (Kingma & Ba, 2014) optimizer, and separately tuned dropout rates for the RNN and word embeddings.
This gives results comparable to the original paper.
All models are trained for up to 50 epochs, batch size 32 , a ﬁxed learning rate of 0 . 001 , and hidden dimensionality of 128 .
We use a dropout of 0 .
5 for input word embeddings, 0 .
2 for SRU layers, and 0 .
3 for LSTM layers.
Results Table 3 summarizes our results and state-of-the-art results as of December, 2017.
LSTM models achieve 69 .
6% exact match and 78 .
9% F1 score.
These results are comparable to the original work (Chen et al., 2017).
SRU models achieve 70 .
3% exact match and 79 .
5% F1 score, outperforming the LSTM models.
Moreover, SRU exhibits 6x to 10x speed-up, more than 69% reduction in total training time.
The most recent state-of-the-art methods both use RNNs to encode text, and can potentially beneﬁt from our approach.
4. L ANGUAGE M ODELING
Dataset We use the Penn Treebank corpus (PTB).
The processed data and splits are taken from
Mikolov et al. (2010).
The data contains about 1M tokens with a truncated vocabulary of 10k.
Following standard practice, the training data is treated as a long sequence split to mini batches, the models are trained using truncated back-propagation-through-time (BPTT; Williams & Peng, 1990).
Setup We largely follow the conﬁguration of prior work (Zaremba et al., 2014; Gal & Ghahramani, 2016; Zoph & Le, 2016).
We use a batch size of 32 and truncated back-propagation with 35 steps.
The dropout probability is 0 .
75 for the input embedding and output softmax layer.
The standard dropout and variational dropout probability are 0 .
for stacked RNN layers.
We use SGD with an initial learning rate of 1 . and gradient clipping.
We train up to 300 epochs, and start to decrease the learning rate by a factor of 0.98 after 175 epochs.
We use the identity activation function for g ( · ) .
[Table]
Results Table 4 shows perplexity results.
We use a parameter budget of 24 million for a fair comparison.
cuDNN LSTM obtains a perplexity of with 73 - 79 seconds per epoch.
This result is worse than most prior work.
We attribute this difference to the lack of variational dropout support in the cuDNN implementation.
SRU obtains better perplexity compared to cuDNN LSTM and prior work, reaching 64 .
7 with three recurrent layers and 60 .
3 with six layers.
4 SRU also provides better speed-perplexity trade-off: training a six-layer RNN takes 47 seconds per epoch.
4. M ACHINE T RANSLATION
Dataset We use the WMT 2014 English → German translation task.
We pre-process the training corpus following standard practice (Peitz et al., 2014; Li et al., 2014; Jean et al., 2015).
About 4M translation pairs are left after processing.
The news-test-2014 data is used as the test set and the concatenation of news-test-2012 and news-test-2013 is used as the development set.
Setup We use OpenNMT (Klein et al., 2017), and extend the Pytorch version 5 with our SRU implementation.
OpenNMT uses a seq2seq model in a recurrent encoder-decoder architecture with attention (Luong et al., 2015).
By default, the model provides h t − 1 , the hidden state of decoder at step t − 1 , as input to step t .
Although this can potentially improve translation quality, it impedes parallelization and slows down training.
We disable this option.
All models are trained with hidden state and word embedding size of 500 , 15 epochs, SGD with initial learning rate of 1 . 0 , and batch size 64 .
We modify the default of OpenNMT, and use a dropout rate of 0 . 1 and a weight decay of 10 − 5 .
This leads to better results for both RNN implementations.
Results Table 5 shows the translation results.
We obtain better BLEU scores compared to the reports OpenNMT results (Klein et al., 2017).
SRU with 10 stacked layers achieves a BLEU score of 20 . 7 while cuDNN LSTM achieves 20 .
45 using more parameters and more training time.
SRU scales better, and we can stack many layers of SRU without signiﬁcant time increase.
Each additional SRU layer in encoder and decoder adds four minutes per training epoch, while adding an LSTM layer adds 23 minutes.
In comparison, the other operations (e.g., attention and output softmax) take aboutminutes.
We do not observe over-ﬁtting on the development set, even when using 10 layers.
[Table]
BLEU
[Table]
RNNs
Klein et al. [Table] Klein et al. (2017) + BPE
[Table]
cuDNN LSTM (wd = 0 )
[Table]
cuDNN LSTM (wd = 10 − 5 ) [Table] Our setup cuDNN LSTM
[Table]
cuDNN LSTM [Table] cuDNN
[Table]
State
[Table]
of
[Table]
the-art Results
GNMT Ensemble (Wu et al., 2016b)
[Table]
ConvS2S Ensemble (Gehring et al., 2017)
[Table]
Transformer (Vaswani et al., 2017)
[Table]
Transformer (Vaswani et al., 2017) 213m 28.
[Table]
We also include the most recent state-of-the-art results in Table 5.
In our experiments, we use the default model provided by OpenNMT as our baseline architecture.
Both RNN and non-RNN architectures have achieved signiﬁcantly better BLEU scores on the En-Ge translation task than the
OpenNMT default model.
Combining SRU with state-of-the-art architectures such as Transformer is an important research direction, which we plan to explore in future work.
For instance, with SRU only 4 minutes are spent per recurrent layer.
This reduction in RNN time enables introducing recurrent computations into the Transformer architecture.
4. S PEECH R ECOGNITION
Dataset We use the Switchboard-1 corpus (Godfrey et al., 1992).
The training data includes about 300 hours of speech from 4 , 870 sides of conversations between 520 speakers.
The test data includes about two hours of speech from 40 sides of conversations from the 2000 Hub5 evaluation.
Setup We use Kaldi (Povey et al., 2011) for feature extraction, decoding, and training of initial
HMM-GMM models.
We use the standard Kaldi recipes to train maximum likelihood-criterion context-dependent speaker adapted acoustic models with Mel-Frequency Cepstral Coefﬁcient (MFCC).
We apply forced alignment to generate labels for neural network acoustic model training.
We use the Computational Network Toolkit (CNTK; Yu et al., 2014) instead of PyTorch.
We experiment with uni-directional and bi-directional models, with and without state-level Minimum Bayes
Risk (sMBR) training (Kingsbury et al., 2012).
We use a tri-gram based language model instead of a
RNN based language model on top of the acoustic model.
See Appendix B for the complete setup details.
Results Table 6 summarizes the results.
CNTK uses a special batching algorithms for RNNs, and hence we were not able to use our customized SRU kernel.
However, even without any kernel optimization, the SRU is faster than an LSTM using the same number of parameters.
SRU also achieves better results on WER.
Adding the highway connections (Srivastava et al., 2015) to the
LSTM performs slightly worse than the baseline.
Removing the dependency on the internal state h in [Table] the LSTM can improve the speed but causes a slight decrease in performance.
C includes further experiments with different highway structures and number of layers
C ONCLUSION
We present Simple Recurrent Unit (SRU), a recurrent architecture that is as fast as CNN and easily scales to over 10 layers.
Our evaluation on a variety of NLP and speech recognition tasks demonstrates the effectiveness of SRU.
We open source our implementation to facilitate future research.
A C OMPARISON OF M ODEL V ARIANTS AND Q UASI -RNN
SRU and Quasi-RNN (Bradbury et al., 2017) both were developed with the goal of speeding up the computation of recurrent neural networks.
The Quasi-RNN design aims to combine k -gram convolutions with adaptive pooling (i.e., fo-pooling ) instead of traditional order-oblivious pooling, such as max pooling or average pooling.
Similar to other convolutional architectures (Kalchbrenner et al., 2014; Kim, 2014; Wang et al., 2015; Dauphin et al., 2017), a k -gram ﬁlter width > 1 is used throughout the experiments reported.
While Quasi-RNN is based on adding light recurrence to convolutional network, SRU is an instance of the recurrent architectures introduced by Lei et al.
While this prior work focused on theoretical characteristics of the networks and their generalization, we focus on the implementation details of SRU, including practical optimizations, and on its applicability to a wide range of tasks.
The
SRU computation is similar to the degenerate case of setting k = 1 in the k -gram ﬁlter in Quasi-RNN.
The computation is then reduced to a simple feed-forward transformation (matrix multiplication), and loses the properties and quality of a convolution.
The SRU architecture and results we present also differ from Quasi-RNN in key technical decisions:
• CUDA Optimization: While the Quasi-RNN implementation of fo-pooling is done as a CUDA kernel function.
The rest of the element-wise computation is performed via separate function calls on top of the software library (e
g., Chainer).
This choice does not allow this type of architecture to achieve its full potential.
In contrast, we implement element-wise fusion to enable further speed optimization (Section 2.3).
• Activation and Highway Connections: SRU and Quasi-RNN differ in how the non-linear activation function is applied.
Quasi-RNN follows the common practice with convolutional models, where the non-linear activation is applied with the convolution operation
before pooling (i.e., ˜ x
= tanh ( W ∗ x ) ).
In contrast, in SRU the activation function g ( · ) is applied to compute the internal state (i.e., h t = g ( c t ) ) following the derivation in (Lei et al., 2017).
SRU also includes highway connections in the architecture for better generalization of deep networks.
Effect of Element-wise Fusion We compare the speed of the fo-pooling implementation used in Quasi-RNN and the fused kernel implementation used in SRU.
We implement a version of unidirectional and bi-directional SRU that uses fo-pooling and separate element-wise function calls in PyTorch.
Figure 3 (left) presents the speed comparison on ﬁve classiﬁcation tasks (Section 4.1) and SQuAD (Section 4.2).
The fused element-wise kernel achieves 24% to 94% speed improvement across the six benchmarks.
Effect of Activation and Highway Connection We compare the performance of SRU variants with different activation functions and Quasi-RNN with ﬁlter width k = 1 on the classiﬁcation and
SQuAD datasets.
For the classiﬁcation tasks, we train using using the Adam optimizer with default 0 .
001 learning rate, 0 weight decay, and dropout probability tuned from values { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 } .
We perform three trials for 10 -fold cross validation for each model and dataset.
We report the average test accuracy of model conﬁgurations that achieve the best development results.
For SQuAD, we train all models for a maximum of 100 epochs using the Admax optimizer with learning rate 0.001.
We perform three independent trials, and report the average performance.
Figure 3 (right), Table 4, and Table 5 summarize the results:
• SRU performs at least as good as Quasi-RNN, and often outperforms it.
This illustrates the advantages of the design choices in SRU, including the highway connections and activation implementation.
• We observe that the optimal choice of activation function varies depending on the task.
For example, we found ReLU activation to work best on the classiﬁcation benchmarks, but the identity activation performs the best on SQuAD, while ReLU performs worst.
• The effect of highway connections is best highlighted in the question answering dataset.
Without the highway connections, we observe a performance decrease when stacking more than four recurrent layers (Table 5).
In contrast, adding highway connections results in > 1% absolute improvement of exact match score.
Figure 3 (right) also shows that
no
Table 1 CR 0.235834699 SUBJ 0.2364684346 MR 0.2977398607 TREC 0.
0.2977398607 0.6810370494 0.70914788 0.9431818182 MPQA 0.70914788 SQuAD 0.
C R
S U B J
M R
T R E C
M P Q A
S Q u A D
[Figure]
[Figure]
Model
CR
SUBJ
MR
TREC
MPQA
Quasi-RNN ( k = 1 ) 83.6 ± 2.
93.3 ± 0.
81.6 ± 1.
92.7 ± 0.
89.6 ± 1.
Quasi-RNN ( k = 1 ) + highway 84.0 ± 1.
93.4 ± 0.
82.1 ± 1.
93.2 ± 0.
89.6 ± 1.
SRU (identity) 84.1 ± 1.
93.5 ± 0.
82.1 ± 1.
93.8 ± 0.
89.7 ± 1.
SRU (tanh) 84.2 ± 1.
93.5 ± 0. 82.1 ± 1. 93.9 ± 0. [Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
82.5 ± 1.
[Figure]
[Figure]
[Figure]
Number of recurrent layers
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
over-ﬁtting occurs on the development set within 100 training epochs.
This suggests that models with highway connections are likely to generalize better.
B S PEECH R ECOGNITION E XPERIMENTAL S ETUP D ETAILS
Following Sainath et al.
, all weights are randomly initialized from the uniform distribution with range [ − 0 .
05] , and all biases are initialized to 0 without generative or discriminative pre-training (Seide et al., 2011).
All neural network models, unless noted otherwise, are trained with a cross-entropy criterion using truncated BPTT.
No momentum is used for the ﬁrst epoch, and a momentum of 0 .
9 is used for subsequent epochs (Zhang et al., 2015).
We apply an L 2 constraint regularization with weight 10 − 5 (Hinton et al., 2012) .
We experiment with uni-directional and bi-directional models.
To train the uni-directional model, we unroll 20 frames and use 80 utterances in each mini-batch.
We also delayed the output of the
LSTM and SRU by 10 frames as suggested by Sak et al.
(2014) to add more context.
To train the bidirectional model, we use the latency-controlled method described in Zhang et al. (2015).
We set N c = 80 and N r = 20 and processed 40 utterances simultaneously.
In addition to our vanilla model, we also experiment state-level Minimum Bayes Risk (sMBR) training (Kingsbury et al., 2012).
To train the recurrent model with the sMBR criterion, we adopted the two-forward-pass method described by Zhang et al.
and processed 40 utterances simultaneously.
The input features for all models are 80 -dimensional log Mel ﬁlterbank features computed every ten milliseconds, with an additional 3-dimensional pitch features.
The output targets are 8802 -contextdependent triphone states, of which the numbers are determined by the last HMM-GMM training stage.
Our setup can potentially improve if we incorporate some recent techniques that are applicable to
SRU.
For example, LF-MMI for sequence training, i-vectors for speaker adaptation, and speaker perturbation for data augmentation were applied by Povey et al.
All of these techniques can also been used for SRU.
Moreover, different highway variants such as grid LSTM (Hsu et al., 2016) can also further boost our model.
C A DDITIONAL S PEECH R ECOGNITION A NALYSIS
Baseline To identify the LSTM baseline used in Section 4.5, we experiment with varying the number of layers and parameters.
Table 7 shows the performance of different settings.
We follow the setup of Sak et al.
The best LSTM baseline is using ﬁve layers with 1024 units in each layer.
[Table]
Effect of Highway Transform for SRU
The dimensionality of the input x t and h t must be equal in the computation of h t (Equation 5).
However, when this is not the case, for example as in the ﬁrst layer of the SRU, we can use a linear projection W lh to match the dimensions at layer l .
The modiﬁed version of Equation 5 will then be:
[Equation]
We can also use a square matrix W lh for every layer.
Table 8 shows that adding this transformation signiﬁcantly reduces the word error rate from 12 .
6% to 11 .
8% when using the same number of parameters.
This transformation is outside the recurrent loop, and can be parallelized to be computed efﬁciently.
[Table]
Effect of Depth for SRU We study the effect of SRU depth on performance.
7 Table 9 shows word error rate for SRU with different number of layers.
The SRU model outperforms the LSTM model with 10 layers and the same number of parameters, while provide a 1.4x speed-up, even though it is using a non-optimized implementation in CNTK.
The speed gains are mainly a result of requiring less matrix multiplications.
The best performance is achieved with 12 layers
[Table]
in CNTK.
No CUDA-level optimizations are performed.