A comprehensive analysis on attention models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Sequence-to-sequence attention-based models are a promising approach for end-toend speech recognition.
The increased model power makes the training proceduremore difﬁcult, and analysing failure modes of these models becomes harder becauseof the end-to-end nature.
In this work, we present various analyses to betterunderstand training and model properties.
We investigate on pretraining variantssuch as growing in depth and width, and their impact on the ﬁnal performance,which leads to over 8% relative improvement in word error rate.
For a betterunderstanding of how the attention process works, we study the encoder outputand the attention energies and weights.
Our experiments were performed on
Switchboard, LibriSpeech and Wall Street Journal.
Introduction
The encoder-decoder framework with attention [Bahdanau et al., 2014, Luong et al., 2015, Wu et al.,2016] has been successfully applied to automatic speech recognition (ASR)
[Chan et al., 2015, Chiuet al., 2017, Toshniwal et al., 2018, Krishna et al., 2018, Zeyer et al., 2018b, Zeghidour et al., 2018,
Weng et al., 2018, Sabour et al., 2018] and is a promising end-to-end approach.
The model outputsare words, sub-words or characters, and training the model can be done from scratch without anyprerequisites except the training data in terms of audio features with corresponding transcriptions.
In contrast to the conventional hybrid hidden Markov models (HMM) / neural network (NN) approach[Bourlard and Morgan, 1994, Robinson, 1994], the encoder-decoder model does not model thealignment explicitly.
In the hybrid HMM/NN approach, a latent variable of hidden states is introduced,which model the phone state for any given time position.
Thus by searching for the most probablesequence of hidden states, we get an explicit alignment.
There is no such hidden latent variable inthe encoder decoder model.
Instead of the encoder decoder approach.
In some of these models, the attention process is not soft, but ahard decision.
In some of these models, this hard decision becomes a latent variable, such that weinclude several choices in the beam search.
This is also referred to as hard attention.
Examples ofdirectly applying this idea on the usual attention approach are given by Raffel et al.
[2017], Aharoniand Goldberg [2016], Chiu and Raffel [2017], Luo et al.
[2017], Lawson et al.
We study recurrent encoder decoder models in this work, which use long short-term memory (LSTM)units [Hochreiter and Schmidhuber, 1997].
Recently the transformer model [Vaswani et al., 2017]gained attention, which only uses feed-forward and self-attention layers, and the only recurrence isthe label feedback in the decoder.
As this does not include any temporal information, some positionalencoding is added.
This is not necessary for a RNN model, as it can learn such encoding by itself.
We will later show that our LSTM encoder in fact does learn such a representation.
We study attention models in more detail here.
We are interested in when, why and how they failand do an analysis on the search errors and relative error positions.
We study the implicit alignmentbehaviour via the attention weights and energies.
We also analyse the encoder output representationwhich is the base for the attention and ﬁnd that it contains information about the relative position andthat it specially marks frames which should not be attended to, which correspond to silence.
Related work
Belinkov and Glass [2017] analyses the hidden activations of the DeepSpeech
[Amodei et al., 2016]
CTC end-to-end system and shows their correlation to a phoneme frame alignment.
A number of saliency methods [Simonyan et al., 2014, Luisa M Zintgraf and Welling, 2017] are usedfor interpreting model decisions for image classiﬁcation tasks by using the back propagated classscores with respect to the input image.
Erhan et al.
[2009] demonstrate several techniques to visualizeand interpret high-level features of models trained on vision datasets.
Attention plots were used before to understand the behavior of the model [Chorowski et al., 2015]. Beck et al. [2018b] performed a comparison of the alignment behavior between hybrid HMM/NNmodels, the inverted HMM and attention models.
ASR tasks and baselines
In all cases, we use the RETURNN framework [Zeyer et al., 2018a] for neural network training andinference, which is based on TensorFlow [TensorFlow Development Team, 2015] and contains somecustom CUDA kernels.
In case of the attention models, we also use RETURNN for decoding.
Insome cases, the feature extraction, and in the hybrid case the decoding, is performed with RASR[Wiesler et al., 2014].
All used conﬁgs as well as used source code are published.
3.1 Switchboard 300h
The Switchboard corpus [Godfrey et al., 2003] consists of English telephone speech.
We use the300h train dataset (LDC97S62), and a 90% subset for training, and a small part for cross validation,which is used for learning rate scheduling and to select a few models for decoding.
We decode andreport WER on Hub5’00 and Hub5’01.
We use Hub5’00 to select the best model which we report thenumbers on.
Our hybrid HMM/NN model uses a deep bidirectional LSTM as described in Zeyer et al.
Ourbaseline has 6 layers with 500 nodes in each direction.
It uses dropout of 10% on the non-recurrentinput of each LSTM layer, gradient noise with standard deviation of 0.3, Adam with Nesterovmomentum (Nadam)
[Dozat, 2015], Newbob learning rate scheduling [Zeyer et al., 2017], and focalloss
[Lin et al., 2017].
Our attention model uses byte pair encoding [Sennrich et al., 2015] as subword units.
We follow thebaseline with about 1000 BPE units as descriped in Zeyer et al.
[2018b].
All our baselines and a comparison to results from the literature are summarized in Table 1.
[Table] [Table]
other
[Table] [Table]
[Liptchinsky [Table] [Table]
[Zeyer et al., 2018b]
[Table]
[Zeyer et al
[Table] 1000h [Table] The LibriSpeech dataset [Panayotov et al., 2015a] are read audio books and consists of about 1000h ofspeech.
A subset of the training data is used for cross-validation, to perform learning rate schedulingand to select a number of models for full decoding.
We use the dev-other set for selecting the ﬁnalbest model.
The end-to-end attention model uses byte pair encoding (BPE)
[Sennrich et al., 2015] as subwordunits with a vocabulary of 10k BPE units. We follow the baseline as descriped in Zeyer et al. [2018b]. A comparison of our baselines and other models are in Table 2.
[Table]
3.3 Wall Street Journal
The Wall Street Journal (WSJ) dataset [Paul and Baker, 1992] is read text from the WSJ.
We use90% of si284 for training, the remaming for cross validation and learning rate scheduling, dev93 forvalidation and selection of the ﬁnal model, and eval92 for the ﬁnal evaluation.
We trained an end-to-end attention model using BPE subword units, with a vocabulary size of about1000 BPE units.
Our preliminary results are shown in Table 3.
Our attention model is based on theimproved pretraining scheme as described in Section 5.
Error analysis
We analyse the errors in the decoding process during beam search.
In Fig.
1a we collected thecorrespondence between the beam size and the WER or the amount of search errors.
We just countthe search errors where the models recognized sentence (via beam search) has a worse model scorethan the ground truth sentence.
We observe that we do only very few search errors, and the amount ofsearch errors seems independent from the ﬁnal WER performance.
Thus we conclude that we mostlyhave a problem in the model.
We also were interested in the score difference between the best recognized sentence and the groundtruth sentence.
The results are in Fig.
1b.
We can see that they concentrate on the lower side, around10%, which is an indicator why a low beam size is sufﬁcient.
We analysed the errors which are made w.r.t.
the relative position in the sentence.
TODO plot errorrelative occurance: in WER of best seq for att model.
TODO error relative occurance: relative beamsearch error.
(also ref to [Mimura et al., 2018], they have a similar plot.)
We observed that moreerrors are made at the very beginning of the sentence.
This is in contrast to hybrid HMM/NN models.
TODO plot error relative occurance in WER of best seq for hybrid model.
This might be due tothe left-to-right (forward) factorization.
The attention process likely works better given a correcthistory of words.
This property of attention models leads to the idea of combining a forward andbackward decoder, as it was done in Mimura et al.
[2018], where the improvement likely stems fromthis observation.
Analysis on pretraining
It has been observed that pretraining can be substantial for good performance, and sometimes to get aconverging model at all [Zeyer et al., 2018a,b].
We provide a study on cases where pretraining is [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
Beam Search Error Analysis [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] mandatory to get to convergence, and compare the performance with and without pretraining. All theexperiments were performed with attention models. The pretraining variant of the Switchboard baseline (6 layers, time reduction 8 after pretraining)consists of these steps: 1. starts with 2 layers (layer 1 and 6), time reduction 32, and dropout aswell as label smoothing disabled; 2.
enable dropout; 3.
3 layers (layer 1, 2 and 6); 4.
4 layers (layer1, 2, 3 and 6); 5. 5 layers (layer 1, 2, 3, 4 and 6); 6.
all 6 layers; 7. decrease time reduction to 8;8.
ﬁnal model, enable label smoothing.
Each pretrain step is repeated for 5 epochs, where one epochcorresponds to 1/6 of the whole train corpus.
In addition, a linear learning rate warmup is performedfrom 1e-4 to 1e-3 in 10 epochs.
We have to start with 2 layers as we want to have the time pooling inbetween the LSTM layers.
In Table 4, performed on Switchboard, we varried the number of encoderlayers and encoder LSTM units, both with and without pretraining.
We observe that the overal bestmodel is with 4 layers.
and without pretraining in this pretraining variant.
I.e. we showed that wecan directly start with 4 layers and time reduction 8 and yield very good results.
We even can startdirectly with 6 layer with a reduced learning rate.
This was surprising to us, as this was not possiblein earlier experiments.
This might be due to more stable hyper parameters.
We note that overal all thepretraining experiments seems to run more stable.
We also can see that with 6 layers (and also more),pretraining yields better results than no pretraining. factor [Table] [Table]
These results motivated us to perform further investigations into different variants of pretraining.
Itseems that pretraining allows to train deeper model and helps for the convergence in that case, butusing too much pretraining can also hurt.
We showed that we can directly start with a deeper encoderand lower time reduction.
If we still want to train a 6 layer encoder, we maybe could directly startwith more than 2 layers.
In Table 5, we analysed the optimal start number of layers, and the start timereduction.
We observed that starting with a deeper network improves the overal performance, butalso it still helps to then go deeper during pretraining.
Also, at some point, starting too deep doesnot work well.
We also observed that directly starting with time reduction 8 also works and furtherimproves the ﬁnal performance, but it seems that this makes the training slightly more unstable, sothat only starting with 4 layers and time reduction 8 works, but not starting deeper.
In further experiments, we directly start with 4 layers and time reduction 8.
We were also interestedin the optimal number of repetitions of each pretrain step, i.e. how much epochs to train with eachpretrain step; the baseline had 5 repetitions.
We collected the results in Table 6.
In further experiments,we keep 5 repetitions as the default.
It has already be shown by Zeyer et al.
[2018b] that a lower ﬁnal time reduction was optimal.
So far,the lowest time reduction was 8 in our experiments.
By having a pool size of 3 in the ﬁrst time maxpooling layer, we wanted to explore a total time reduction factor of 6.
We can see a comparison in
Table 7.
We can conclude that time reduction 6 performs better.
So far we always kept the top layer (layer 6) during pretraining as our intuition was that it might helpto get always the same time-reduction factor as an input to this layer.
When directly starting with thelow time reduction, we do not need this scheme anymore, and we can always add a new layer on top.
Comparisons are collected in Table 7.
We can conclude that this simpler scheme to add layers on topperforms better.
We also did experiments with growing the encoder width / number of LSTM units during pretraining.
We do this orthogonal to the growing in depth / number of layers.
As before, our ﬁnal number of
LSTM units in each direction of the bidirectional deep LSTM encoder is 1024.
Initially, we start with50% of the ﬁnal width, i.e. with 512 units.
In each step, we linearly increase the number of units suchthat we have the ﬁnal number of units in the last step.
We keep the weights of existing units, andweights from/to newly added units are randomly initialized.
We also decrease the dropout rate by thesame factor.
We collected results in Table 7.
We can see that this width growing scheme performsbetter.
This leads us to our current best model.
Our ﬁndings are that pretraining is in general more stable, esp.
for deep models.
However, thepretraining scheme is important, and less pretraining can improve the performance, although itbecomes more unstable.
We also used the same ﬁnal pretrain conﬁg for WSJ and showed that it alsoworks there.
[Table] [Table]
[Figure] Encoder dimensions visualized using PCA      (b)
Analysis on training variance
We have observed that training attention models can be unstable, and careful tuning of initial learningrate, warmup and pretraining is important.
Related to that, we observe a high training variance.
I.e.with the same conﬁguration but different random seeds, we get some variance in the ﬁnal WERperformance.
Due to non-determinism in some of the used algorithms, we observed this even forthe same random seed.
The variance due to the non-determinism seems to be about the same as dueto random seeds, which is higher than we expected.
Note that it also depends a lot on other hyperparameters.
The hyper parameters for this baseline are relatively stable.
However, with more unstablesettings we have observed some higher variance, and even sometimes some models diverge whileothers converge with the same settings.
We also compare that to hybrid HMM/LSTM models.
It canbe observed that it is lower compared to the attention model.
We argue that is due to the more difﬁcultoptimization problem, and also due to the much bigger model.
All the results can be seen in Table 8.
Analysis of the encoder output
The encoder creates a high-level representation of the input.
It also arguably represents furtherinformation needed for the decoder to know where to attend to.
We try to analyse the output of theencoder and identify and examine the learned function.
One hypothesis for an important function of the encoder is the detection of frames which should notbe attended on by the decoder, i.e. which are silent .
By performing a dimensionality reduction onthe encoder output, which is typically very high-dimensional.
Principal component analysis (PCA)is a method to perform this reduction by projecting the sample points to a subspace deﬁned by then largest principal components.
TODO formalize PCA/T-SNE ?
TODO plot of PCA/T-SNE [-To further verify this behaviour, we chose Switchboard-
 [Table]-]We
[Figure]
E n c od e r d i m e n s i on s have a wonder • ful ea • ster <
/s >  − 0 .0 .0 .
Intuitively, the encoder has to know where it currently is with respect to the input sequence.
Thuswe try to identify individual cells in the LSTM which fulﬁll this purpose of encoding the positionalinformation .
In constrast to the detection of silence described above, we approach this differently byaveraging the encoder outputs across the whole dataset.
This allows us to average out potential errorsas well as identify possible trends.
By qualitatively inspecting the different neurons activations, we have identiﬁed multiple neuronswhich perform the hypothesized function as shown in ﬁgure...
TODO ﬁg relative position neuron.
TODO plot
TODO detects slience.
TODO more?
We observed that the attention weights are always very local in the encoder frames, and oftenfocus mostly on a single encoder frame.
The sharp behaviour in the converged attention weightdistribution has been observed before [Chan et al., 2015].
This can be seen in TODO ref to plotwith att energies.
We conclude that the information about the label is very local in the encoderoutput.
To support this observation, we performed experiments where we explicitly allow only alocal ﬁxed-size window of non-zero attention weights around the arg max of the attention energies,to understand how much we can restrict the local context.
The results can be seen in Table 9. [Table]
This conﬁrms the hypothesis that the information is localized in the encoder.
We explain the gap in performancedue to decoder frames where the model is unsure to attend,and where it helps the decoder to gather information frommultiple frames at once.
TODO plot where att is smeared.plot att weights + energies.
We also observed that in some cases, the attention weightson the encoder start and end frame become active.
Thisseems to be the case when the decoder is unsure.
TODOplot (or visible in prev plot?).
TODO early in training, concentrates on single frame.
WSJ exps, SWB base.6l.red6.pretrain-start4lred6-below-grow3.run2keep.
Conclusion
By our improved pretraining scheme, we improved our baseline by over 8% relative in WER. Weanalysed the encoder output and identiﬁed the representation of the relative position as well as silence.