A comprehensive analysis on attention models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Sequence-to-sequence attention-based models are a promising approach for end-toend speech recognition.
The increased model power makes the training proceduremore difﬁcult, and analysing failure modes of these models becomes harder becauseof the end-to-end nature.
In this work, we present various analyses to betterunderstand training and model properties.
We analyse different patterns of errorsand the effect on the amount of training data.
We investigate on pretraining variantssuch as growing in depth and width, and their impact on the ﬁnal performance,which leads to over 8% improvement in word error rate.
We also highlight the highvariance in different training runs.
For a better understanding of how the attentionprocess works, we study the encoder output and the attention energies and weights.
Our experiments were performed on Switchboard and LibriSpeech.
Introduction
The encoder-decoder framework with attention [Bahdanau et al., 2014, Luong et al., 2015, Wu et al.,2016] has been successfully applied to automatic speech recognition (ASR) [Chan et al., 2015, Chiuet al., 2017, Toshniwal et al., 2018, Krishna et al., 2018, Zeyer et al., 2018b, Zeghidour et al., 2018,
Weng et al., 2018, Sabour et al., 2018] and is one promising end-to-end approach, i.e. the modeloutputs are words, sub-words or characters, and training the model can be done from scratch withoutany prerequisites except the audio features and transcriptions.
In contrast to the conventional hybrid hidden Markov models (HMM) / neural network (NN) approach[Bourlard and Morgan, 1994, Robinson, 1994], the encoder-decoder model does not model thealignment explicitly.
In the hybrid HMM/NN approach, a latent variable of hidden states is introduced,which model the phone state for any given time position.
Thus by searching for the most probablesequence of hidden states, we get an explicit alignment.
There is no such hidden latent variablein the encoder decoder model.
Instead there is the attention process which can be interpret as animplicit soft alignment.
As this is only implicit and soft, it is harder to enforce constraints such asmonotonicity, i.e. that the attention of future label outputs will focus also only to future time frames.
Also, the interpretation of the attention weights as a soft alignment might not be completely valid, asnothing in the model enforces this, i.e. the neural network could learn to pass over information in anypossible way. Thus, the studying the behaviour of the attention model becomes harder. There are other end-to-end approaches as well. Connectionist temporal classiﬁcation [Graves et al.
,2006] has often been applied to ASR in the past [Graves and Jaitly, 2014, Hannun et al., 2014, Miaoet al., 2015, Amodei et al., 2016, Soltau et al., 2017, Audhkhasi et al., 2017, Krishna et al., 2018, Zenkel et al., 2018, Zhang and Lei, 2018]. Other approaches are e.g. the inverted hidden Markov /segmental encoder-decoder model [Doetsch et al., 2017, Beck et al., 2018a], the recurrent transducer[Rao et al., 2017, Battenberg et al., 2017, Prabhavalkar et al., 2017], or the recurrent neural aligner[Sak et al., 2017, Dong et al., 2018].
TODO also stochastic attention. TODO LSTM hochreiter ref. TODO RNN vs transformer. We want to study attention models in more detail here.
We are interested in when, why and how theyfail.
We study the implicit alignment behaviour via the attention weights and energies.
Also, it hasbeen observed that they usually perform comparable to hybrid models with enough training data, butthey perform much worse with too less training data.
We also tried to analyse the internal hiddenrepresentation of the encoder which is the base for the attention. We publish all the conﬁgs to train the models of this work.
Related work
Belinkov and Glass [2017] analyses the hidden activations {+of the DeepSpeech 
 [Amodei et al., 2016] CTC end-to-end system and shows their correlation to a phoneme frame alignment.+} TODO what else?
Baselines
In all cases, we use the RETURNN framework [Zeyer et al., 2018a] for neural network training andinference. In case of the attention models, we also use RETURNN for decoding. In some cases,the feature extraction, and in the hybrid case the decoding, is performed with RASR [Wiesler et al. All used conﬁgs as well as used source code are published.
3.1 Switchboard 300h
The Switchboard corpus [Godfrey et al., 2003] consists of English telephone speech.
We use the300h train dataset (LDC97S62), and a 90% subset for training, and a small part for cross validation,which is used for learning rate scheduling and to select a few models for decoding.
We decode andreport WER on Hub5’00 and Hub5’01.
We use Hub5’00 to select the best model which we report thenumbers on.
Our hybrid HMM/NN model uses a deep bidirectional LSTM as described in Zeyer et al.
Ourbaseline 2 has 6 layers with 500 nodes in each direction.
It uses dropout of 10% on the non-recurrentinput of each LSTM layer, gradient noise with standard deviation of 0.3, Adam with Nesterovmomentum (Nadam)
[Dozat, 2015], Newbob learning rate scheduling [Zeyer et al., 2017], and focalloss
[Lin et al., 2017].
Our attention model uses byte pair encoding [Sennrich et al., 2015] as subword units.
We follow thebaseline with about 1000 BPE units as descriped in Zeyer et al.
[2018b].
All our baselines and a comparison to results from the literature are summarized in Table 1.
3.2 LibriSpeech 1000h
LibriSpeech
TODO
ref.
The end-to-end attention model uses byte pair encoding [Sennrich et al., 2015] as subword units.
Wefollow the baseline as descriped in Zeyer et al.
[2018b].
3.3 Wall Street Journal
The Wall Street Journal (WSJ) dataset [Paul and Baker, 1992] is read text from the WSJ.
We use90% of si284 for training, the remaming for cross validation and learning rate scheduling, dev93 forvalidation and selection of the ﬁnal model, and eval92 for the ﬁnal evaluation.
We trained an end-to-end attention model using BPE subword units, with a vocabulary size of about1000 BPE units.
Our results are preliminary.
We basically applied our currently best Switchboardsetup.
[Table]
[Table] [Table] [Table] [Figure]
Error analysis
We analyse the errors in the decoding process during beam search. In Figure 1 we collected thecorrespondence between the beam size and the WER or the amount of search errors. We just countthe search errors where the models found sentence (via beam search) has a lower model score thanthe ground truth sentence. We also were interested in the score difference between the best found sentence and the ground truthsentence.
The results are in Figure 2.
We can see that they concentrate on the lower side, around 10%,which is an indicator why a low beam size is sufﬁcient.
[Figure]
We analysed the errors which are made w.r.t.
the relative position in the sentence.
TODO plot erroroccurance: in WER of best seq for att model. TODO error occurance: relative beam search error. Weobserved that more errors are made at the very beginning of the sentence.
This is in contrast to hybrid HMM/NN models. TODO plot error relative occurance in WER of best seq for hybrid model.
Thismight be due to the left-to-right (forward) factorization.
The attention process likely works bettergiven a correct history of words.
This property of attention models leads to the idea of combining aforward and backward decoder, as it was done in Mimura et al. [2018], where the improvement likelystems from this observation.
Analysis on the amount of training data
It has been observed that with enough training data, attention models can perform as good as hybridmodels, or at least comparable [Chiu et al., 2018, Zeyer et al., 2018b].
With less training data, thisdoes not seem to be the case anymore.
To understand the performance degradation in relation to the amount of training data, we use
LibriSpeech
[Panayotov et al., 2015] and select subsets with various amounts of training data andtrain systems on each subset.
TODO plot of train amount vs WER.
TODO number of beam search errors vs train amount?
maybe in the same plot.
TODO interestingalign plots for different train amount?
TODO what else?
Analysis on pretraining
It has been observed that pretraining can be substantial for good performance, and sometimes to get aconverging model at all [Zeyer et al., 2018a,b]. We provide a study on cases where pretraining ismandatory to get to convergence, and compare the performance with and without pretraining. All theexperiments were performed with attention models.
The pretraining variant of the Switchboard baseline (6 layers, time reduction 8 after pretraining)consists of these steps: 1. starts with 2 layers (layer 1 and 6), time reduction 32, and dropout as well as label smoothingdisabled2. enable dropout 3 layers (layer 1, 2 and 6)4.
4 layers (layer 1, 2, 3 and 6) enabled or disabled. Trained on Switchboard 300h, using the baseline attention conﬁg. 1 We had to lower the initial learning rate to allow the model to converge.
TODO where is 1?
∗ TODO .
+ TODO .
[Table]
[Table]
[Table]
[Table]
Each pretrain step is repeated for 5 epochs.
In addition, a linear learning rate warmup is performedfrom 1e-4 to 1e-3 in 10 epochs.
We have to start with 2 layers as we want to have the time pooling inbetween the LSTM layers.
In Table 2, performed on Switchboard, we varried the number of encoderlayers and encoder LSTM units, both with and without pretraining.
We observe that the overal bestmodel is with 4 layers.
and without pretraining in this pretraining variant.
I.e. we showed that wecan directly start with 4 layers and time reduction 8 and yield very good results.
We even can startdirectly with 6 layer with a reduced learning rate.
This was surprising to us, as this was not possiblein earlier experiments.
This might be due to more stable hyper parameters.
We note that overal all thepretraining experiments seems to run more stable.
We also can see that with 6 layers (and also more),pretraining yields better results than no pretraining.
These results motivated us to perform further investigations into different variants of pretraining.
Itseems that pretraining allows to train deeper model and helps for the convergence in that case, butusing too much pretraining can also hurt.
We showed that we can directly start with a deeper encoderand lower time reduction.
If we still want to train a 6 layer encoder, we maybe could directly startwith more than 2 layers. In Table 3, we analysed the optimal start number of layers, and the start timereduction. We observed that starting with a deeper network improves the overal performance, butalso it still helps to then go deeper during pretraining. Also, at some point, starting too deep doesnot work well.
We also observed that directly starting with time reduction 8 also works and furtherimproves the ﬁnal performance, but it seems that this makes the training slightly more unstable, sothat only starting with 4 layers and time reduction 8 works, but not starting deeper.
In all cases, the pretraining scheme ends up with a 6 layer encoder, and time reduction factor 8.
[Table] [Table] In further experiments [Table]
we directly start with 4 layers and time reduction 8. We were also interestedin the optimal number of repetitions of each pretrain step, i [Table] e. how much epochs to train with eachpretrain step; the baseline had 5 repetitions.
Note that 1 epoch corresponds to 1/6 of the total trainingdataset in the setup.
We collected the results in Table 4.
In further experiments, we keep 5 repetitionsas the default.
It has already be shown by Zeyer et al.
[2018b] that a lower ﬁnal time reduction was optimal. So far,the lowest time reduction was 8 in our experiments. By having a pool size of 3 in the ﬁrst time maxpooling layer, we wanted to explore a total time reduction factor of 6. We can see a comparison in
Table 5.
We can conclude that time reduction 6 performs better.
So far we always kept the top layer (layer 6) during pretraining as our intuition was that it might helpto get always the same time-reduction factor as an input to this layer.
When directly starting with thelow time reduction, we do not need this scheme anymore, and we can always add a new layer on top.
Comparisons are collected in Table 5.
We can conclude that this simpler scheme to add layers on topperforms better.
We also did experiments with growing the encoder width / number of LSTM units during pretraining.
We do this orthogonal to the growing in depth / number of layers.
As before, our ﬁnal number of
LSTM units in each direction of the bidirectional deep LSTM encoder is 1024.
Initially, we start with50% of the ﬁnal width, i.e. with 512 units.
In each step, we linearly increase the number of units suchthat we have the ﬁnal number of units in the last step.
We keep the weights of existing units, andweights from/to newly added units are randomly initialized.
We also decrease the dropout rate by thesame factor.
We collected results in Table 5.
We can see that this width growing scheme performsbetter.
This leads us to our current best model.
[Table]
[Table]
[Table]
[Table]
[Table]
random [Table]  Analysis on training variance We have observed that training attention models can be unstable, and careful tuning of initial learningrate, warmup and pretraining is important.
Related to that, we observe a high training variance.
I.e. with the same conﬁguration but different random seeds, we get some variance in the ﬁnal WER performance.
We collected results in Table 6a.
Due to non-determinism in some of the usedalgorithms, we observed this even for the same random seed, compare Table 6b. The variance due tothe non-determinism seems to be about the same as due to random seeds, which is higher than weexpected.
Note that it also depends a lot on other hyper parameters. The hyper parameters for thisbaseline are relatively stable. However, we have observed with more unstable settings some highervariance, and even sometimes some models diverge while others converge.
We also compare that to hybrid HMM/LSTM models. The performance variance via different randomseeds can be seen in Table 7a. It can be observed that it is lower compared to the attention model. Weargue that is due to the more difﬁcult optimization problem and also due to the much bigger model
[Table] [Table]  Analysis of the encoder output
The encoder creates a high-level representation of the input.
It also arguably represents furtherinformation needed for the decoder to know where to attend to.
We try to analyse the output of theencoder and identify possible learned features.
TODO plot
TODO detects slience.
TODO relative position neuron.
TODO more?
Analysis of the attention weights
TODO maybe move before the prev section?
We observed that the attention weights are always very local in the encoder frames, and often focusmostly on a single encoder frame.
The sharp behaviour in the converged attention weight distributionhas been observed before [Chan et al., 2015]. We aim to further understand this behaviour.
We performed experiments where we explicitly allow only a local ﬁxed-size window of non-zeroattention weights around the arg max of the attention energies, to understand how much we canrestrict the local context.
Furthermore, we then continue training with this restriction allowing themodel to adapt to the limited encoder access.
TODO [Beck et al., 2018b] ITG paper...
TODO plot att weights.
TODO plot att energies.
TODO smeared when unsure.
sometimes start/end frame active.
att energies look different then.
TODO ...
We also observed that in some cases, the attention weights on the encoder start and end frame becomeactive.
This seems to be the case when the model is unsure. TODO early in training, concentrates on single frame.
WSJ exps, SWB base.6l.red6.pretrain-start4lred6-below-grow3.run2keep.
TODO ... [Table]
Conclusion
TODO
Acknowledgments
TODO