A comprehensive analysis on attention models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Sequence-to-sequence attention-based models are a promising approach for end-toend speech recognition.
The increased model power makes the training proceduremore difﬁcult, and analysing failure modes of these models becomes harder becauseof the end-to-end nature.
In this work, we present various analyses to betterunderstand training and model properties.
We analyse different patterns of errorsand the effect on the amount of training data.
We investigate on pretraining variantssuch as growing in depth and width, and their impact on the ﬁnal performance,which leads to over 8% improvement in word error rate.
We also highlight the highvariance in different training runs.
For a better understanding of how the attentionprocess works, we study the encoder output and the attention energies and weights.
Our experiments were performed on Switchboard and LibriSpeech.
Introduction
The encoder-decoder framework with attention [Bahdanau et al., 2014, Luong et al., 2015, Wu et al.,2016] has been successfully applied to speech recognition [Chan et al., 2015, Chiu et al., 2017,
Toshniwal et al., 2018, Krishna et al., 2018, Zeyer et al., 2018b] and is one promising end-to-endapproach, i.e. the models output are words, sub-words or characters, and training the model can bedone from scratch without any prerequisites except the audio features and transcriptions. In contrast to the conventional hybrid hidden Markov models (HMM) / neural network (NN) approach[Bourlard and Morgan, 1994, Robinson, 1994], the encoder-decoder model does not model thealignment explicitly. In the hybrid HMM/NN approach, a latent variable of hidden states is introduced,which model the phone state for any given time position. Thus by searching for the most probablesequence of hidden states, you get an explicit alignment.
There is no such hidden latent variablein the encoder decoder model. Instead there is the attention process which can be interpret as animplicit soft alignment. As this is only implicit and soft, it is harder to enforce constraints such asmonotonicity, i.e. that the attention of future label outputs will focus also only to future time frames. Also, the interpretation of the attention weights as a soft alignment might not be completely valid, asnothing in the model enforces this, i.e. the neural network could learn to pass over information in anypossible way.
Thus, the studying the behaviour of the attention model becomes harder.
We want to study attention models in more detail here.
We are interested in when, why and how theyfail.
We study the implicit alignment behaviour via the attention weights and energies.
Also, it hasbeen observed that they usually perform comparable to hybrid models with enough training data, butthey perform much worse with too less training data. We also tried to analyse the internal hiddenrepresentation of the encoder which is the base for the attention.  Related work
Belinkov and Glass [2017] provides an analysis of the hidden activations TODO
[Figure] [Figure] TODO order of the sections ok? general structure of this paper...
[Figure]
Error analysis
TODO errors appear mostly at beginning, in contrast to hybrid models.
TODO [Mimura et al., 2018]
Mimura et al.
: "Forward-Backward Attention Decoder" improves maybe because of this.
TODOerror occurance: in WER of best seq.
also in beam search error.
TODO include hybrid model inhistogram / side-by-side
TODO mention beam size.
TODO try also for high beam size, where other people already observedthat WER increases again. enabled or disabled.
1 We had to lower the initial learning rate to allow the model to converge.
[Table] TODO more here?
or not own section for this?
TODO Histogramm über die Score-Abweichungen zwischen bester erkannter und korrekter Hypothese, relativ zur Score-Variation innerhalb des Beams, jeweils für Attention und ggf.
Hybrid.
Analysis on the amount of training data
It has been observed that with enough training data, attention models can perform as good as hybridmodels, or at least comparable [Chiu et al., 2018, Zeyer et al., 2018b].
With less training data, thisdoes not seem to be the case anymore.
To understand the performance degradation in relation to the amount of training data, we use
LibriSpeech
[Panayotov et al., 2015] and select subsets with various amounts of training data andtrain systems on each subset.
TODO plot of train amount vs WER.
TODO number of beam search errors vs train amount?
maybe in the same plot.
TODO interestingalign plots for different train amount?
TODO what else?
Analysis on pretraining
It has been observed that pretraining can be substantial for good performance, and sometimes to get aconverging model at all [Zeyer et al., 2018a,b].
We provide a study on cases where pretraining is mandatory to get to convergence, and compare theperformance with and without pretraining.
[Table] In Table 1... TODO
TODO what else do we have?
tried various number of repetitions?
Analysis on training variance
We have observed that training attention models can be unstable, and careful tuning of initial learningrate, warmup and pretraining is important.
Related to that, we observe a high training variance.
I.e.with the same conﬁguration but different random seeds, we get a high variance in the ﬁnal WERperformance.
We collected results in Table 2.
Due to non-determinism in some of the used algorithms, we observed this even for the same randomseed.
TODO tables...
TODO comparison to hybrid train variance.
Analysis of the encoder output
The encoder creates a high-level representation of the input.
It also arguably represents furtherinformation needed for the decoder to know where to attend to.
We try to analyse the output of theencoder and identify possible learned features.
TODO plot
TODO detects slience.
TODO relative position neuron.
TODO more?
Analysis of the attention weights
TODO maybe move before the prev section?
We observed that the attention weights are always very local in the encoder frames, and often focusmostly on a single encoder frame.
The sharp behaviour in the converged attention weight distribution has been observed before [Chanet al., 2015], we aim to further understand this behaviour.
We performed experiments where we explicitly allow only a local ﬁxed-size window of non-zeroattention weights around the arg max of the attention energies, to understand how much we canrestrict the local context.
Furthermore, we then continue training with this restriction allowing themodel to adapt to the limited encoder access.
TODO [Beck et al., 2018] ITG paper...
TODO plot att weights.
TODO plot att energies.
TODO smeared when unsure.
sometimes start/end frame active.
att energies look different then. [Table]
TODO [Table]
We also observed that in some cases, the attention weights on the encoder start and end frame becomeactive.
This seems to be the case when the model is unsure.
TODO ...
Conclusion
TODO
Acknowledgments
TODO