A comprehensive analysis on attention models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Sequence-to-sequence attention-based models are a promising approach for end-toend speech recognition.
The increased model power makes the training proceduremore difﬁcult, and analysing failure modes of these models becomes harder becauseof the end-to-end nature.
In this work, we present various analyses to betterunderstand training and model properties.
We analyse different patterns of errorsand the effect on the amount of training data.
TODO remove? We investigateon pretraining variants such as growing in depth and width, and their impacton the ﬁnal performance, which leads to over 8% relative improvement in worderror rate.
We also highlight the high variance in different training runs.
For abetter understanding of how the attention process works, we study the encoderoutput and the attention energies and weights.
Our experiments were performed on Switchboard, LibriSpeech and Wall Street Journal.
Introduction
The encoder-decoder framework with attention [Bahdanau et al., 2014, Luong et al., 2015, Wu et al.,2016] has been successfully applied to automatic speech recognition (ASR)
[Chan et al., 2015, Chiuet al., 2017, Toshniwal et al., 2018, Krishna et al., 2018, Zeyer et al., 2018b, Zeghidour et al., 2018,
Weng et al., 2018, Sabour et al., 2018] and is a promising end-to-end approach. The model outputsare words, sub-words or characters and training the model can be done from scratch without anyprerequisites except the training data in terms of audio features with corresponding transcriptions.
In contrast to the conventional hybrid hidden Markov models (HMM) / neural network (NN) approach[Bourlard and Morgan, 1994, Robinson, 1994], the encoder-decoder model does not model thealignment explicitly.
In the hybrid HMM/NN approach, a latent variable of hidden states is introduced,which model the phone state for any given time position.
Thus by searching for the most probablesequence of hidden states, we get an explicit alignment.
There is no such hidden latent variable inthe encoder decoder model.
Instead there is the attention process which can be interpreted as animplicit soft alignment.
As this is only implicit and soft, it is harder to enforce constraints such asmonotonicity, i.e. that the attention of future label outputs will focus also only to future time frames.
Also, the interpretation of the attention weights as a soft alignment might not be completely valid, asthe encoder itself can shift around and reorder evidence, i.e. the neural network could learn to passover information in any possible way. E.g. the encoder could compress all the information of theinput into a single frame and the decoder can learn to just attend on this single frame. We observedthis behaviour in early stages of the training.
Thus, studying the temporal "alignment" behaviour ofthe attention model becomes more difﬁcult.
There are other end-to-end approaches as well.
Connectionist temporal classiﬁcation [Graves et al.
,2006] has often been applied to ASR in the past [Graves and Jaitly, 2014, Hannun et al., 2014, Miaoet al., 2015, Amodei et al., 2016, Soltau et al., 2017, Audhkhasi et al., 2017, Krishna et al., 2018,
Zenkel et al., 2018, Zhang and Lei, 2018].
Other approaches are e.g. the inverted hidden Markov / segmental encoder-decoder model [Doetsch et al., 2017, Beck et al., 2018a], the recurrent transducer [Rao et al., 2017, Battenberg et al., 2017, Prabhavalkar et al., 2017], or the recurrent neural aligner[Sak et al., 2017, Dong et al., 2018].
Depending on the interpretation, these can all be seen as variantsof the encoder decoder approach.
In some of these models, the attention process is not soft, but ahard decision.
In some of these models, this hard decision becomes a latent variable, such that weinclude several choices in the beam search.
This is also referred to as hard attention.
Examples ofdirectly applying this idea on the usual attention approach are given by Raffel et al.
[2017], Aharoniand Goldberg [2016], Chiu and Raffel [2017], Luo et al.
[2017], Lawson et al.
We study recurrent encoder decoder models in this work, which use long short-term memory (LSTM)units [Hochreiter and Schmidhuber, 1997].
Recently the transformer model [Vaswani et al., 2017]gained attention, which only uses feed-forward and self-attention layers, and the only recurrence isthe label feedback in the decoder.
As this does not include any temporal information, some positionalencoding is added.
This is not necessary for a RNN model, as it can learn such encoding by itself.
We will later show that our LSTM encoder in fact does learn such a representation.
We study attention models in more detail here.
We are interested in when, why and how they failand do an analysis on the search errors and relative error positions.
We study the implicit alignmentbehaviour via the attention weights and energies.
We also analyse the encoder output representationwhich is the base for the attention and ﬁnd that it contains information about the relative position andthat it specially marks frames which should not be attended to, which correspond to silence.
Related work
Belinkov and Glass [2017] analyses the hidden activations of the DeepSpeech
[Amodei et al., 2016]
CTC end-to-end system and shows their correlation to a phoneme frame alignment.
A number of saliency methods [Simonyan et al., 2014, Luisa M Zintgraf and Welling, 2017] are usedfor interpreting model decisions for image classiﬁcation tasks by using the back propagated classscores with respect to the input image.
Erhan et al.
[2009] demonstrate several techniques to visualizeand interpret high-level features of models trained on vision datasets.
TODO what else?
ASR tasks and baselines
In all cases, we use the RETURNN framework [Zeyer et al., 2018a] for neural network training andinference, which is based on TensorFlow [TensorFlow Development Team, 2015] and contains somecustom CUDA kernels.
In case of the attention models, we also use RETURNN for decoding.
Insome cases, the feature extraction, and in the hybrid case the decoding, is performed with RASR[Wiesler et al., 2014].
All used conﬁgs as well as used source code are published.
3.1 Switchboard 300h
The Switchboard corpus [Godfrey et al., 2003] consists of English telephone speech.
We use the300h train dataset (LDC97S62), and a 90% subset for training, and a small part for cross validation,which is used for learning rate scheduling and to select a few models for decoding.
We decode andreport WER on Hub5’00 and Hub5’01.
We use Hub5’00 to select the best model which we report thenumbers on.
Our hybrid HMM/NN model uses a deep bidirectional LSTM as described in Zeyer et al.
Ourbaseline 2 has 6 layers with 500 nodes in each direction.
It uses dropout of 10% on the non-recurrentinput of each LSTM layer, gradient noise with standard deviation of 0.3, Adam with Nesterovmomentum (Nadam)
[Dozat, 2015], Newbob learning rate scheduling [Zeyer et al., 2017], and focalloss
[Lin et al., 2017].
Our attention model uses byte pair encoding [Sennrich et al., 2015] as subword units.
We follow thebaseline with about 1000 BPE units as descriped in Zeyer et al.
[2018b].
[Table]
[Table] [Table] [Table] All our baselines and a comparison to results from the literature are summarized in Table 1.
3.2 LibriSpeech 1000h
The LibriSpeech dataset [Panayotov et al., 2015a] are read audio books and consists of about 1000h ofspeech.
A subset of the training data is used for cross-validation, to perform learning rate schedulingand to select a number of models for full decoding.
We use the dev-other set for selecting the ﬁnalbest model.
The end-to-end attention model uses byte pair encoding (BPE)
[Sennrich et al., 2015] as subwordunits with a vocabulary of 10k comparison and other models are in Table
[Table]
The Wall Street Journal (WSJ) dataset [Paul and Baker, 1992] is read text from the WSJ.
We use90% of si284 for training, the remaming for cross validation and learning rate scheduling, dev93 forvalidation and selection of the ﬁnal model, and eval92 for the ﬁnal evaluation.
We trained an end-to-end attention model using BPE subword units, with a vocabulary size of about1000 BPE units.
Our preliminary results are shown in Table 3.
Error analysis
We analyse the errors in the decoding process during beam search.
In Fig. 1a we collected thecorrespondence between the beam size and the WER or the amount of search errors.
We just countthe search errors where the models recognized sentence (via beam search) has a worse model scorethan the ground truth sentence.
We observe that we do only very few search errors, and the amount ofsearch errors seems independent from the ﬁnal WER performance.
Thus we conclude that we mostlyhave a problem in the model.
We also were interested in the score difference between the best recognized sentence and the groundtruth sentence.
The results are in Fig. 1b.
We can see that they concentrate on the lower side, around10%, which is an indicator why a low beam size is sufﬁcient.
We analysed the errors which are made w.r.t.
the relative position in the sentence.
TODO plot errorrelative occurance: in WER of best seq for att model.
TODO error relative occurance: relative beamsearch error.
(also ref to [Mimura et al., 2018], they have a similar plot.)
We observed that moreerrors are made at the very beginning of the sentence.
This is in contrast to hybrid HMM/NN models.
TODO plot error relative occurance in WER of best seq for hybrid model.
This might be due tothe left-to-right (forward) factorization.
The attention process likely works better given a correcthistory of words.
This property of attention models leads to the idea of combining a forward andbackward decoder, as it was done in Mimura et al.
[2018], where the improvement likely stems fromthis observation.
Analysis on pretraining
It has been observed that pretraining can be substantial for good performance, and sometimes to get aconverging model at all [Zeyer et al., 2018a,b].
We provide a study on cases where pretraining ismandatory to get to convergence, and compare the performance with and without pretraining.
All theexperiments were performed with attention models.
[Table] [Table] The pretraining variant of the Switchboard baseline (6 layers, time reduction 8 after pretraining)consists of these steps:
1. starts with 2 layers (layer 1 and 6), time reduction 32, and dropout as well as label smoothing
disabled
enable dropout
3 layers (layer 1, 2 and 6)
4 layers (layer 1, 2, 3 and 6)
5 layers (layer 1, 2, 3, 4 and 6)
all 6 layers
decrease time reduction to
ﬁnal model.
enable label smoothing
Each pretrain step is repeated for 5 epochs.
In addition, a linear learning rate warmup is performedfrom 1e-4 to 1e-3 in 10 epochs.
We have to start with 2 layers as we want to have the time pooling inbetween the LSTM layers.
In Table 4, performed on Switchboard, we varried the number of encoderlayers and encoder LSTM units, both with and without pretraining.
We observe that the overal bestmodel is with 4 layers.
and without pretraining in this pretraining variant.
I.e. we showed that wecan directly start with 4 layers and time reduction 8 and yield very good results.
We even can startdirectly with 6 layer with a reduced learning rate.
This was surprising to us, as this was not possiblein earlier experiments.
This might be due to more stable hyper parameters.
We note that overal all thepretraining experiments seems to run more stable.
We also can see that with 6 layers (and also more),pretraining yields better results than no pretraining.
These results motivated us to perform further investigations into different variants of pretraining.
Itseems that pretraining allows to train deeper model and helps for the convergence in that case, butusing too much pretraining can also hurt.
We showed that we can directly start with a deeper encoderand lower time reduction.
If we still want to train a 6 layer encoder, we maybe could directly startwith more than 2 layers.
In Table 5, we analysed the optimal start number of layers, and the start timereduction.
We observed that starting with a deeper network improves the overal performance, butalso it still helps to then go deeper during pretraining.
Also, at some point, starting too deep doesnot work well.
We also observed that directly starting with time reduction 8 also works and furtherimproves the ﬁnal performance, but it seems that this makes the training slightly more unstable, sothat only starting with 4 layers and time reduction 8 works, but not starting deeper.
In further experiments, we directly start with 4 layers and time reduction 8.
We were also interestedin the optimal number of repetitions of each pretrain step, i.e. how much epochs to train with eachpretrain step; the baseline had 5 repetitions.
Note that 1 epoch corresponds to 1/6 of the total trainingdataset in the setup.
We collected the results in Table 6.
In further experiments, we keep 5 repetitionsas the default.
depth and width with scheme enabled or disabled. We had to lower the initial learning rate to allow the model to converge: [Table] [Table]
[Table]
repetitions
[Table]
[Table]
[Table]
[Table]
It has already be shown by Zeyer et al.
[2018b] that a lower ﬁnal time reduction was optimal.
So far,the lowest time reduction was 8 in our experiments.
By having a pool size of 3 in the ﬁrst time maxpooling layer, we wanted to explore a total time reduction factor of 6.
We can see a comparison in
Table 7.
We can conclude that time reduction 6 performs better.
So far we always kept the top layer (layer 6) during pretraining as our intuition was that it might helpto get always the same time-reduction factor as an input to this layer.
When directly starting with thelow time reduction, we do not need this scheme anymore, and we can always add a new layer on top.
Comparisons are collected in Table 7.
We can conclude that this simpler scheme to add layers on topperforms better.
We also did experiments with growing the encoder width / number of LSTM units during pretraining.
We do this orthogonal to the growing in depth / number of layers.
As before, our ﬁnal number of
LSTM units in each direction of the bidirectional deep LSTM encoder is 1024.
Initially, we start with50% of the ﬁnal width, i.e. with 512 units.
In each step, we linearly increase the number of units suchthat we have the ﬁnal number of units in the last step.
We keep the weights of existing units, andweights from/to newly added units are randomly initialized.
We also decrease the dropout rate by thesame factor.
We collected results in Table 7.
We can see that this width growing scheme performsbetter.
This leads us to our current best model.
Our ﬁndings are that pretraining is in general more stable, esp. for deep models. However, thepretraining scheme is important, and less pretraining can improve the performance, although it [Table]
[Table]
σ 0. becomes more unstable.
We also used the same ﬁnal pretrain conﬁg for WSJ and showed that it alsoworks there.
We have observed that training attention models can be unstable, and careful tuning of initial learningrate, warmup and pretraining is important.
Related to that, we observe a high training variance.
I.e. with the same conﬁguration but different random seeds, we get some variance in the ﬁnal
WER performance.
We collected results in Table 8a.
Due to non-determinism in some of the usedalgorithms, we observed this even for the same random seed, compare Table 8b.
The variance due tothe non-determinism seems to be about the same as due to random seeds, which is higher than weexpected.
Note that it also depends a lot on other hyper parameters.
The hyper parameters for thisbaseline are relatively stable.
However, we have observed with more unstable settings some highervariance, and even sometimes some models diverge while others converge.
We also compare that to hybrid HMM/LSTM models.
The performance variance via different randomseeds can be seen in Table 9a.
It can be observed that it is lower compared to the attention model.
Weargue that is due to the more difﬁcult optimization problem, and also due to the much bigger model.
In Table 9b we can see the performance variance due to non-deterministic algorithms.
Analysis of the encoder output
TODO font size is wrong
TODO maybe just side-by-side for switchboard/librispeech using TSNE (or PCA?)
The encoder creates a high-level representation of the input.
It also arguably represents furtherinformation needed for the decoder to know where to attend to.
We try to analyse the output of theencoder and identify and examine the learned function.
One hypothesis for an important function of the encoder is the detection of frames which are silent.
By performing a dimensionality reduction on the encoder output, which is typically very highdimensional.
Principal component analysis (PCA) is a method to perform this reduction by projectingthe sample points to a subspace deﬁned by the n largest principal components.
TODO formalize
PCA/T-SNE ?
TODO {+plot of PCA/T-SNE To further verify this behaviour, we chose Switchboard-
 [Table] [Table] E n c od e r ti m e s t+}
e p s
Encoder visualization for ’switchboard-1/sw02340B/sw2340B-ms98-a-0096’ as a dataset for two primary reasons.
First, we have a good baseline and second, for the 300-hourdataset there are phonetic alignments available from a hybrid model.
This allows us to conﬁrm thehypothesis by computing the percentage in which the attention model has correctly identiﬁed thesilence frames.
TODO cite swb
[Figure]
Intuitively, the encoder has to know where it currently is with respect to the input sequence.
Thuswe try to identify individual cells in the LSTM which fulﬁll this purpose of encoding the positionalinformation.
In constrast to the detection of silence described above, we approach this differently byaveraging the encoder outputs across the whole dataset.
This allows us to average out potential errorsas well as identify possible trends.
By qualitatively inspecting the different neurons activations, we have identiﬁed multiple neuronswhich perform the hypothesized function as shown in ﬁgure... TODO ﬁg relative position neuron. TODO plot TODO detects slience.
TODO more?
[Figure] E n c od e r d i m e n s i on s 0 . [Figure] Encoder output combined attention weights
[Figure] E n c od e r d i m e n s i on s 0 .
[Figure] [Figure] [Figure] We observed that the attention weights are always very local in the encoder frames, and often focusmostly on a single encoder frame.
The sharp behaviour in the converged attention weight distributionhas been observed before [Chan et al., 2015].
This can be seen in TODO ref to plot with att energies. We conclude that the information about the label is very local in the encoder output.
To support thisobservation, we performed experiments where we explicitly allow only a local ﬁxed-size window ofnon-zero attention weights around the arg max of the attention energies, to understand how much wecan restrict the local context.
Furthermore, we then continue training with this restriction allowing themodel to adapt to the limited encoder access.
The results can be seen in Table 10.
This conﬁrms thehypothesis that the information is localized in the encoder.
We explain the small gap in performancedue to decoder frames where the model is unsure to attend, and where it helps the decoder to gather
[Figure]
[Figure] information from multiple frames at once.
TODO plot where att is smeared.
plot att weights +energies.
TODO [Beck et al., 2018b] ITG paper... Model
Window size Word error rate [%] no training after further training baseline
[Table] global) 19. local (global argmax) 20. – local (argmax) 164.
[Table]
We also observed that in some cases, the attention weights on the encoder start and end frame becomeactive.
This seems to be the case when the decoder is unsure.
TODO plot (or visible in prev plot?). TODO early in training, concentrates on single frame. WSJ exps, SWB base.6l.red6.pretrain-start4lred6-below-grow3.run2keep.
Conclusion
TODO