A comprehensive analysis on attention models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Sequence-to-sequence attention-based models are a promising approach for end-toend speech recognition.
The increased model power makes the training proceduremore difﬁcult, and analyzing failure modes of these models becomes harder becauseof the end-to-end nature.
In this work, we present various analyses to betterunderstand training and model properties.
We investigate on pretraining variantssuch as growing in depth and width, and their impact on the ﬁnal performance,which leads to over 8% relative improvement in word error rate.
For a betterunderstanding of how the attention process works, we study the encoder outputand the attention energies and weights.
Our experiments were performed on
Switchboard, LibriSpeech and Wall Street Journal.
Introduction
The encoder-decoder framework with attention [Bahdanau et al., 2014, Luong et al., 2015, Wu et al.,2016] has been successfully applied to automatic speech recognition (ASR)
[Chan et al., 2015, Chiuet al., 2017, Toshniwal et al., 2018, Krishna et al., 2018, Zeyer et al., 2018b, Zeghidour et al., 2018,
Weng et al., 2018, Sabour et al., 2018] and is a promising end-to-end approach.
The model outputsare words, sub-words or characters, and training the model can be done from scratch without anyprerequisites except the training data in terms of audio features with corresponding transcriptions.
In contrast to the conventional hybrid hidden Markov models (HMM) / neural network (NN) approach[Bourlard and Morgan, 1994, Robinson, 1994], the encoder-decoder model does not model thealignment explicitly.
In the hybrid HMM/NN approach, a latent variable of hidden states is introduced,which model the phone state for any given time position.
Thus by searching for the most probablesequence of hidden states, we get an explicit alignment.
There is no such hidden latent variable inthe encoder decoder model.
Instead there is the attention process which can be interpreted as animplicit soft alignment.
As this is only implicit and soft, it is harder to enforce constraints such asmonotonicity, i.e. that the attention of future label outputs will focus also only to future time frames.
Also, the interpretation of the attention weights as a soft alignment might not be completely valid, asthe encoder itself can shift around and reorder evidence, i.e. the neural network could learn to passover information in any possible way. E.g. the encoder could compress all the information of theinput into a single frame and the decoder can learn to just attend on this single frame. We observedthis behavior in early stages of the training. Thus, studying the temporal "alignment" behavior of theattention model becomes more difﬁcult.
Other end-to-end models such as connectionist temporal classiﬁcation [Graves et al., 2006] has oftenbeen applied to ASR in the past [Graves and Jaitly, 2014, Hannun et al., 2014, Miao et al., 2015,
Amodei et al., 2016, Soltau et al., 2017, Audhkhasi et al., 2017, Krishna et al., 2018, Zenkel et al.
,2018, Zhang and Lei, 2018].
Other approaches are e.g. the inverted hidden Markov / segmentalencoder-decoder model [Doetsch et al., 2017, al., 2018a], the recurrent transducer [Rao al.
,2017, Battenberg et al., 2017, Prabhavalkar et al., 2017], or the recurrent neural aligner
[Sak et al.
,2017, Dong al., 2018].
Depending on the interpretation, these can all be seen as variants of the encoder decoder approach.
In some of these models, the attention process is not soft, but a harddecision.
This hard decision can also become a latent variable such that we include several choices inthe beam search.
This is also referred to as hard attention.
Examples of directly applying this idea onthe usual attention approach are given by Raffel et al.
[2017], Aharoni and Goldberg [2016], Chiuand Raffel [2017], Luo et al.
[2017], Lawson et al.
We study recurrent NN (RNN) encoder decoder models in this work, which use long short-termmemory (LSTM) units [Hochreiter and Schmidhuber, 1997].
Recently the transformer model[Vaswani et al., 2017] gained attention, which only uses feed-forward and self-attention layers, andthe only recurrence is the label feedback in the decoder.
As this does not include any temporalinformation, some positional encoding is added.
This is not necessary for a RNN model, as it canlearn such encoding by itself.
We will later show that our LSTM encoder in fact does learn such arepresentation.
We study attention models in more detail here.
We are interested in when, why and how they failand do an analysis on the search errors and relative error positions.
We study the implicit alignmentbehavior via the attention weights and energies.
We also analyze the encoder output representationand ﬁnd that it contains information about the relative position and that it specially marks frameswhich should not be attended to, which correspond to silence.
Related work
Belinkov and Glass [2017] analyses the hidden activations of the DeepSpeech
[Amodei et al., 2016]
CTC end-to-end system and shows their correlation to a phoneme frame alignment.
A number of saliency methods [Simonyan et al., 2014, Luisa M Zintgraf and Welling, 2017] are usedfor interpreting model decisions for image classiﬁcation tasks by using the back propagated classscores with respect to the input image.
Erhan et al.
[2009] demonstrate several techniques to visualizeand interpret high-level features of models trained on vision datasets.
Attention plots were used before to understand the behavior of the model [Chorowski et al., 2015].
Beck et al.
[2018b] performed a comparison of the alignment behavior between hybrid HMM/NNmodels, the inverted HMM and attention models.
ASR tasks and baselines
In all cases, we use the RETURNN framework [Zeyer et al., 2018a] for neural network training andinference, which is based on TensorFlow [TensorFlow Development Team, 2015] and contains somecustom CUDA kernels.
In case of the attention models, we also use RETURNN for decoding.
Insome cases, the feature extraction, and in the hybrid case the decoding, is performed with RASR[Wiesler et al., 2014].
All used conﬁgs as well as used source code are published.
3.1 Switchboard 300h
The Switchboard corpus [Godfrey et al., 2003] consists of English telephone speech.
We use the300h train dataset (LDC97S62), and a 90% subset for training, and a small part for cross validation,which is used for learning rate scheduling and to select a few models for decoding.
We decode andreport WER on Hub5’00 and Hub5’01.
We use Hub5’00 to select the best model which we report thenumbers on.
Our hybrid HMM/NN model uses a deep bidirectional LSTM as described in Zeyer et al.
Ourbaseline has 6 layers with 500 nodes in each direction.
It uses dropout of 10% on the non-recurrentinput of each LSTM layer, gradient noise with standard deviation of 0.3, Adam with Nesterovmomentum (Nadam)
[Dozat, 2015], Newbob learning rate scheduling [Zeyer et al., 2017], and focalloss
[Lin et al., 2017].
Our attention model uses byte pair encoding [Sennrich et al., 2015] as subword units.
We follow thebaseline with about 1000 BPE units as described in Zeyer et al.
[2018b].
All our baselines and a comparison to results from the literature are summarized in Table 1.
[Table]
results
[Table]
The LibriSpeech dataset [Panayotov et al., 2015a] are read audio books and consists of about 1000h ofspeech.
A subset of the training data is used for cross-validation, to perform learning rate schedulingand to select a number of models for full decoding.
We use the dev-other set for selecting the ﬁnalbest model.
The end-to-end attention model uses byte pair encoding (BPE)
[Sennrich et al.,
[Table]
3.3 Wall Street Journal
The Wall Street Journal (WSJ) dataset [Paul and Baker, 1992] is read text from the WSJ.
We use90% of si284 for training, the remaining for cross validation and learning rate scheduling, dev93 forvalidation and selection of the ﬁnal model, and eval92 for the ﬁnal evaluation.
We trained an end-to-end attention model using BPE subword units, with a vocabulary size of about1000 BPE units.
Our preliminary results are shown in Table 3.
Our attention model is based on theimproved pretraining scheme as described in Section 5.
Error analysis
We analyze the errors in the decoding process during beam search.
In Fig.
1a we collected thecorrespondence between the beam size and the WER or the amount of search errors.
We just countthe search errors where the models recognized sentence (via beam search) has a worse model scorethan the ground truth sentence.
We observe that we do only very few search errors, and the amount ofsearch errors seems independent from the ﬁnal WER performance.
Thus we conclude that we mostlyhave a problem in the model.
We also were interested in the score difference between the best recognized sentence and the groundtruth sentence.
The results are in Fig.
1b.
We can see that they concentrate on the lower side, around10%, which is an indicator why a low beam size is sufﬁcient.
Analysis on pretraining
It has been observed that pretraining can be substantial for good performance, and sometimes to get aconverging model at all [Zeyer et al., 2018a,b].
We provide a study on cases where pretraining ismandatory to get to convergence, and compare the performance with and without pretraining.
All theexperiments were performed with attention models.
The pretraining variant of the Switchboard baseline (6 layers, time reduction 8 after pretraining)consists of these steps: 1. starts with 2 layers (layer 1 and 6), time reduction 32, and dropout aswell as label smoothing disabled; 2.
enable dropout; 3.
3 layers (layer 1, 2 and 6); 4.
4 layers (layer1, 2, 3 and 6); 5. 5 layers (layer 1, 2, 3, 4 and 6); 6.
all 6 layers; 7. decrease time reduction to 8;8.
ﬁnal model, enable label smoothing.
Each pretrain step is repeated for 5 epochs, where one epochcorresponds to 1/6 of the whole train corpus.
In addition, a linear learning rate warm-up is performedfrom 1e-4 to 1e-3 in 10 epochs.
We have to start with 2 layers as we want to have the time pooling inbetween the LSTM layers.
In Table 4, performed on Switchboard, we varied the number of encoder [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Beam Search Error Analysis
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
layers and encoder LSTM units, both with and without pretraining.
We observe that the overall bestmodel is with 4 layers.
and without pretraining in this pretraining variant.
I.e. we showed that wecan directly start with 4 layers and time reduction 8 and yield very good results.
We even can startdirectly with 6 layer with a reduced learning rate.
This was surprising to us, as this was not possiblein earlier experiments.
This might be due to more stable hyper parameters.
We note that overall allthe pretraining experiments seems to run more stable.
We also can see that with 6 layers (and alsomore), pretraining yields better results than no pretraining.
These results motivated us to perform further investigations into different variants of pretraining.
Itseems that pretraining allows to train deeper model and helps for the convergence in that case, butusing too much pretraining can also hurt.
We showed that we can directly start with a deeper encoderand lower time reduction.
If we still want to train a 6 layer encoder, we maybe could directly startwith more than 2 layers.
In Table 5, we analyzed the optimal start number of layers, and the starttime reduction.
We observed that starting with a deeper network improves the overall performance,but also it still helps to then go deeper during pretraining.
Also, at some point, starting too deep doesnot work well.
We also observed that directly starting with time reduction 8 also works and furtherimproves the ﬁnal performance, but it seems that this makes the training slightly more unstable, sothat only starting with 4 layers and time reduction 8 works, but not starting deeper. repetitions. In all cases, the pretraining scheme ends up with a 6 layer encoder, and time reduction factor 8. [Table]
[Table] LSTM layer [Table] In further experiments, we directly start with 4 layers and time reduction 8.
We were also interestedin the optimal number of repetitions of each pretrain step, i.e. how much epochs to train with eachpretrain step; the baseline had 5 repetitions.
We collected the results in Table 6.
In further experiments,we keep 5 repetitions as the default.
It has already be shown by Zeyer et al.
[2018b] that a lower ﬁnal time reduction was optimal.
So far,the lowest time reduction was 8 in our experiments.
By having a pool size of 3 in the ﬁrst time maxpooling layer, we wanted to explore a total time reduction factor of 6.
We can see a comparison in
Table 7.
We can conclude that time reduction 6 performs better.
So far we always kept the top layer (layer 6) during pretraining as our intuition was that it might helpto get always the same time-reduction factor as an input to this layer.
When directly starting with thelow time reduction, we do not need this scheme anymore, and we can always add a new layer on top.
Comparisons are collected in Table 7.
We can conclude that this simpler scheme to add layers on topperforms better.
We also did experiments with growing the encoder width / number of LSTM units during pretraining.
We do this orthogonal to the growing in depth / number of layers.
As before, our ﬁnal number of
LSTM units in each direction of the bidirectional deep LSTM encoder is 1024.
Initially, we start with50% of the ﬁnal width, i.e. with 512 units.
In each step, we linearly increase the number of units suchthat we have the ﬁnal number of units in the last step.
We keep the weights of existing units, andweights from/to newly added units are randomly initialized.
We also decrease the dropout rate by thesame factor.
We collected results in Table 7.
We can see that this width growing scheme performsbetter.
This leads us to our current best model.
Our ﬁndings are that pretraining is in general more stable, esp.
for deep models.
However, thepretraining scheme is important, and less pretraining can improve the performance, although itbecomes more unstable.
We also used the same ﬁnal pretrain conﬁg for WSJ and showed that it alsoworks there.
Analysis on training variance
We have observed that training attention models can be unstable, and careful tuning of initial learningrate, warm-up and pretraining is important.
Related to that, we observe a high training variance.
I.e. with the same conﬁguration but different random seeds, we get some variance in the ﬁnal WERperformance.
Due to non-determinism in some of the used algorithms, we observed this even forthe same random seed.
The variance due to the non-determinism seems to be about the same as dueto random seeds, which is higher than we expected.
Note that it also depends a lot on other hyperparameters.
The hyper parameters for this baseline are relatively stable.
However, with more unstablesettings we have observed some higher variance, and even sometimes some models diverge whileothers converge with the same settings.
We also compare that to hybrid HMM/LSTM models.
It canbe observed that it is lower compared to the attention model.
We argue that is due to the more difﬁcultoptimization problem, and also due to the much bigger model.
All the results can be seen in Table 8.
[Table]
[Table]
[Figure]
[Figure] [Table] [Figure]  Analysis of the encoder output
The encoder creates a high-level representation of the input.
It also arguably represents furtherinformation needed for the decoder to know where to attend to.
We try to analyze the output of theencoder and identify and examine the learned function.
One hypothesis for an important function of the encoder is the detection of frames which should notbe attended on by the decoder, i e. which are silent .
By performing a dimensionality reduction onthe encoder output, which is typically very high-dimensional.
Principal component analysis (PCA)is a method to perform this reduction by projecting the sample points to a subspace deﬁned by then largest principal components.
We test this hypothesis on Switchboard where we have a phoneticalignment available.
This allows us to conﬁrm the hypothesis by computing the percentage in whichthe attention model has correctly identiﬁed the silence frames.
Intuitively, the encoder has to know where it currently is with respect to the input sequence.
Thuswe try to identify individual cells in the LSTM which fulﬁll this purpose of encoding the positionalinformation .
In contrast to the detection of silence described above, we approach this differently byaveraging the encoder outputs across the whole dataset.
This allows us to average out potential errorsas well as identify possible trends.
By qualitatively inspecting the different neurons activations, wehave identiﬁed multiple neurons which perform the hypothesized function as shown in Fig. 2c.
We observed that the attention weights are always very local in the encoder frames, and often focus mostly on a single encoder frame.
The sharp behavior in the converged attentionweight distribution has been observed before [Chan et al., 2015, Beck et al., 2018b].
Weconclude that the information about the label is very local in the encoder output.
To support this observation, we performed experiments where we explicitly allow only a local ﬁxed- E n c od e r d i m e n s i on s y ea h i ’ d n e v e r t hough t a bou t t h a t w a y i don ’ t r ea ll y l ook i gu e ss
i ough t t
{+o s o m e ti m e s < / s > 0 . 
+}
5.− 0 .
75 − 0 .
50 − 0 .
[Figure]
size window of non-zero attention weights around the arg max of the attention energies, tounderstand how much we can restrict the local context.
The can be seen in Table [Table] ∞ [Table]
local
[Table]
This conﬁrms the hypothesis that the information is localized in the encoder.
We explain the gap in performancedue to decoder frames where the model is unsure to attend,and where it helps the decoder to gather information frommultiple frames at once.
Conclusion
By our improved pretraining scheme, we improved ourbaseline by over 8% relative in WER.
We analyzed the encoder output and identiﬁed the representation of the relativeposition as well as silence.