A comprehensive analysis on attention models
Abstract
Sequence-to-sequence attention-based models are a promising approach for end-toend speech recognition.
The increased model power makes the training procedure more difﬁcult, and analyzing failure modes of these models becomes harder because of the end-to-end nature.
In this work, we present various analyses to better understand training and model properties.
We investigate on pretraining variants such as growing in depth and width, and their impact on the ﬁnal performance, which leads to over 8% relative improvement in word error rate.
For a better understanding of how the attention process works, we study the encoder output and the attention energies and weights.
Our experiments were performed on
Switchboard, LibriSpeech and Wall Street Journal.
Introduction
The encoder-decoder framework with attention [Bahdanau et al., 2015, Luong et al., 2015, Wu et al., 2016] has been successfully applied to automatic speech recognition (ASR)
[Chan et al., 2015, Chiu et al., 2017, Toshniwal et al., 2018, Krishna et al., 2018, Zeyer et al., 2018b, Zeghidour et al., 2018a,
Weng et al., 2018, Sabour et al., 2018] and is a promising end-to-end approach.
The model outputs are words, sub-words or characters and training the model can be done from scratch without any prerequisites except the training data in terms of audio features with corresponding transcriptions.
In contrast to the conventional hybrid hidden Markov models (HMM) / neural network (NN) approach [Bourlard and Morgan, 1994, Robinson, 1994], the encoder-decoder model does not model the alignment explicitly.
In the hybrid HMM/NN approach, a latent variable of hidden states is introduced, which model the phone state for any given time position.
Thus by searching for the most probable sequence of hidden states, we get an explicit alignment.
There is no such hidden latent variable in the encoder decoder model.
Instead there is the attention process which can be interpreted as an implicit soft alignment.
As this is only implicit and soft, it is harder to enforce constraints such as monotonicity, i.e. that the attention of future label outputs will focus also only to future time frames.
Also, the interpretation of the attention weights as a soft alignment might not be completely valid, as the encoder itself can shift around and reorder evidence, i.e. the neural network could learn to pass over information in any possible way.
E.g. the encoder could compress all the information of the input into a single frame and the decoder can learn to just attend on this single frame.
We observed this behavior in early stages of the training.
Thus, studying the temporal "alignment" behavior of the attention model becomes more difﬁcult.
Other end-to-end models such as connectionist temporal classiﬁcation [Graves et al., 2006] has often been applied to ASR in the past [Graves and Jaitly, 2014, Hannun et al., 2014, Miao et al., 2015,
Amodei et al., 2016, Soltau et al., 2017, Audhkhasi et al., 2017, Krishna et al., 2018, Zenkel et al., 2018, Zhang and Lei, 2018].
Other approaches are e.g. the inverted hidden Markov / segmental encoder-decoder model [Doetsch et al., 2017, Beck et al., 2018a], the recurrent transducer [Rao et al., 2017, Battenberg et al., 2017, Prabhavalkar et al., 2017a], or the recurrent neural aligner
[Sak et al.,
2017, Dong et al., 2018].
Depending on the interpretation, these can all be seen as variants of the encoder decoder approach.
In some of these models, the attention process is not soft, but a hard decision.
This hard decision can also become a latent variable such that we include several choices in the beam search.
This is also referred to as hard attention.
Examples of directly applying this idea on the usual attention approach are given by Raffel et al.
[2017], Aharoni and Goldberg [2016], Chiu* and Raffel* [2018], Luo et al.
[2017], Lawson et al.
We study recurrent NN (RNN) encoder decoder models in this work, which use long short-term memory (LSTM) units [Hochreiter and Schmidhuber, 1997].
Recently the transformer model [Vaswani et al., 2017] gained attention, which only uses feed-forward and self-attention layers, and the only recurrence is the label feedback in the decoder.
As this does not include any temporal information, some positional encoding is added.
This is not necessary for a RNN model, as it can learn such encoding by itself, which we demonstrate later for our attention encoder.
We study attention models in more detail here.
We are interested in when, why and how they fail and do an analysis on the search errors and relative error positions.
We study the implicit alignment behavior via the attention weights and energies.
We also analyze the encoder output representation and ﬁnd that it contains information about the relative position and that it specially marks frames which should not be attended to, which correspond to silence.
Related work
Karpathy [2015] analyzes individual neuron activations of a RNN language model and ﬁnds a neuron which becomes sensitive to the position in line.
Belinkov and Glass [2017] analyzed the hidden activations of the DeepSpeech
[Amodei et al., 2016] CTC end-to-end system and shows their correlation to a phoneme frame alignment.
Palaskar and Metze [2018] analyzed the encoder state and the attention weights of an attention model and makes similar observations as we do.
Attention plots were used before to understand the behaviour of the model [Chorowski et al., 2015].
Beck et al.
[2018b] performed a comparison of the alignment behavior between hybrid HMM/NN models, the inverted HMM and attention models.
[Prabhavalkar et al., 2017b] investigate the effects of varying block sizes, attention types, and sub-word units.
Understanding the inner working of a speech recognition system is also subject in [Tang et al., 2017], where the authors examine activation distribution and temporal patterns, focussing on the comparison between LSTM and GRU systems.
A number of saliency methods [Simonyan et al., 2014, Luisa M Zintgraf and Welling, 2017, Sundararajan et al., 2017] are used for interpreting model decisions.
ASR tasks and baselines
In all cases, we use the RETURNN framework [Zeyer et al., 2018a] for neural network training and inference, which is based on TensorFlow [TensorFlow Development Team, 2015] and contains some custom CUDA kernels.
In case of the attention models, we also use RETURNN for decoding.
All experiments are performed on single GPUs, we did not take advantage of multi-GPU training.
In some cases, the feature extraction, and in the hybrid case the decoding, is performed with RASR [Wiesler et al., 2014].
All used conﬁgs as well as used source code are published.
3.1 Switchboard 300h
The Switchboard corpus [Godfrey et al., 2003] consists of English telephone speech.
We use the 300h train dataset (LDC97S62), and a 90% subset for training, and a small part for cross validation, which is used for learning rate scheduling and to select a few models for decoding.
We decode and report WER on Hub5’00 and Hub5’01.
We use Hub5’00 to select the best model which we report the numbers on.
Our hybrid HMM/NN model uses a deep bidirectional LSTM as described by Zeyer et al.
Our baseline has 6 layers with 500 nodes in each direction.
It uses dropout of 10% on the non-recurrent input of each LSTM layer, gradient noise with standard deviation of 0.3, Adam with Nesterov
[Table]
momentum (Nadam)
[Dozat, 2015], Newbob learning rate scheduling [Zeyer et al., 2017], and focal loss [Lin et al., 2017].
Our attention model uses byte pair encoding [Sennrich et al., 2015] as subword units.
We follow the baseline with about 1000 BPE units as described by Zeyer et al.
[2018b].
All our baselines and a comparison to results from the literature are summarized in Table 1.
3.2 LibriSpeech 1000h
The LibriSpeech dataset [Panayotov et al., 2015a] are read audio books and consists of about 1000h of speech.
A subset of the training data is used for cross-validation, to perform learning rate scheduling and to select a number of models for full decoding.
We use the dev-other set for selecting the ﬁnal best model.
The end-to-end attention model uses byte pair encoding (BPE)
[Sennrich et al., 2015] as subword units with a vocabulary of 10k BPE units.
We follow the baseline as described by Zeyer et al.
[2018b].
A comparison of our baselines and other models are in Table 2.
3.3 Wall Street Journal 80h
The Wall Street Journal (WSJ) dataset [Paul and Baker, 1992] is read text from the WSJ.
We use 90% of si284 for training, the remaining for cross validation and learning rate scheduling, dev93 for validation and selection of the ﬁnal model, and eval92 for the ﬁnal evaluation.
We trained an end-to-end attention model using BPE subword units, with a vocabulary size of about 1000 BPE units.
Our preliminary results are shown in Table 3.
Our attention model is based on the improved pretraining scheme as described in Section 5.
Error analysis
We analyze the errors in the decoding process during beam search.
In Fig.
1 we collected the correspondence between the beam size and the WER or the amount of search errors.
We just count
[Table] [Table]
[Panayotov et al.,
[Table] [Table]
[Panayotov et al.,
[Table] [Table]
[Chan and
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
attention [Chan et al., 2016]
[Table]
[Chorowski and Jaitly, 2016]
[Table]
[Chorowski and Jaitly, 2016]
[Table]
[Zhang et al., 2017] quite
[Table]
[Renduchintala et al., 2018]
[Table]
[Sabour et al., 2018]
[Table]
attention this work
SWB
[Table]
this work
[Table]
the search errors where the models recognized sentence (via beam search) has a worse model score than the ground truth sentence.
We observe that we do only very few search errors, and the amount of search errors seems independent from the ﬁnal WER performance.
Thus we conclude that we mostly have a problem in the model.
We also were interested in the score difference between the best recognized sentence and the ground truth sentence.
The results are in Fig.
We can see that they concentrate on the lower side, around 10%, which is an indicator why a low beam size seems to be sufﬁcient.
Analysis on pretraining
It has been observed that pretraining can be substantial for good performance, and sometimes to get a converging model at all [Zeyer et al., 2018a,b].
We provide a study on cases with attention-based models where pretraining beneﬁts convergence and compare the performance with and without pretraining
[Figure] [Figure]
[Table] [Table]
[Table]
The pretraining variant of the Switchboard baseline (6 layers, time reduction 8 after pretraining) consists of these steps: 1.
starts with 2 layers (layer 1 and 6), time reduction 32, and dropout as well as label smoothing disabled; 2.
enable dropout; 3.
3 layers (layer 1, 2 and 6); 4.
4 layers (layer 1, 2, 3 and 6); 5. 5 layers (layer 1, 2, 3, 4 and 6); 6.
all 6 layers; 7. decrease time reduction to 8; 8.
ﬁnal model, enable label smoothing.
Each pretrain step is repeated for 5 epochs, where one epoch corresponds to 1/6 of the whole train corpus.
In addition, a linear learning rate warm-up is performed from 1e-4 to 1e-3 in 10 epochs.
We have to start with 2 layers as we want to have the time pooling in between the LSTM layers.
In Table 4, performed on Switchboard, we varied the number of encoder layers and encoder LSTM units, both with and without pretraining.
We observe that the overall best model is with 4 layers without the pretraining variant.
I.e. we showed that we can directly start withlayers and time reduction 8 and yield very good results.
We even can start directly with 6 layer with a reduced learning rate.
This was surprising to us, as this was not possible in earlier experiments.
This might be due to a reduced and improved BPE vocabulary.
We note that overall all the pretraining experiments seems to run more stable.
We also can see that with 6 layers (and also more), pretraining yields better results than no pretraining.
factor
[Table] [Table]
[Table]
These results motivated us to perform further investigations into different variants of pretraining.
It seems that pretraining allows to train deeper model, however using too much pretraining can also hurt.
We showed that we can directly start with a deeper encoder and lower time reduction.
In Table 5, we analyzed the optimal initial number of layers , and the initial time reduction .
We observed that starting with a deeper network improves the overall performance, but also it still helps to then go deeper during pretraining, and starting too deep does not work well.
We also observed that directly starting with time reduction 8 also works and further improves the ﬁnal performance, but it seems that this makes the training slightly more unstable.
In further experiments, we directly start withlayers and time reduction 8.
We were also interested in the optimal number of repetitions of each pretrain step, i.e. how much epochs to train with each pretrain step; the baseline had 5 repetitions.
We collected the results in Table 6.
In further experiments, we keep 5 repetitions as the default.
It has already been shown by Zeyer et al.
[2018b] that a lower ﬁnal time reduction performed better.
So far, the lowest time reduction was 8 in our experiments.
By having a pool size of 3 in the ﬁrst time max pooling layer, we achieve a better-performing model with time reduction factor of 6 as shown in Table 7.
So far we always kept the top layer (layer 6) during pretraining as our intuition was that it might help to get always the same time reduction factor as an input to this layer.
When directly starting with the low time reduction, we do not need this scheme anymore, and we can always add a new layer on top.
Comparisons are collected in Table 7.
We can conclude that this simpler scheme to add layers on top performs better.
We also did experiments with growing the encoder width / number of LSTM units during pretraining.
We do this orthogonal to the growing in depth / number of layers.
As before, our ﬁnal number of
LSTM units in each direction of the bidirectional deep LSTM encoder is 1024.
Initially, we start with 50% of the ﬁnal width, i.e. with 512 units.
In each step, we linearly increase the number of units such that we have the ﬁnal number of units in the last step.
We keep the weights of existing units, and weights from/to newly added units are randomly initialized.
We also decrease the dropout rate by the same factor.
We can see that this width growing scheme performs better.
This leads us to our current best model.
Our ﬁndings are that pretraining is in general more stable, esp.
for deep models.
However, the pretraining scheme is important, and less pretraining can improve the performance, although it becomes more unstable.
We also used the same improved pretraining scheme and time reductionfor WSJ as well as LibriSpeech and observed similar improvements, compare Table 2 and Table 3.
Analysis on training variance
We have observed that training attention models can be unstable, and careful tuning of initial learning rate, warm-up and pretraining is important.
Related to that, we observe a high training variance.
I.e. with the same conﬁguration but different random seeds, we get some variance in the ﬁnal WER performance.
We observed this even for the same random seed, which we suspect stems from non-deterministic behaviour in TensorFlow operations such as tf.reduce_sum based on kernels
[Table]
[Table] [Table]
sequence.
The differ- ent points are colored according to their relative position, and • non silence frames silence frames.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
using CUDA atomics.
This training variance seems to be about the same as due to random seeds, which is higher than we expected.
Note that it also depends a lot on other hyper parameters.
For example, in a previous iteration of the model using a larger BPE vocabulary, we have observed more unstable training with higher variance, and even sometimes some models diverge while others converge with the same settings.
We also compare that to hybrid HMM/LSTM models.
It can be observed that it is lower compared to the attention model.
We argue that is due to the more difﬁcult optimization problem, and also due to the much bigger model.
All the results can be seen in Table 8.
Analysis of the encoder output
The encoder creates a high-level representation of the input.
It also arguably represents further information needed for the decoder to know where to attend to.
We try to analyze the output of the encoder and identify and examine the learned function.
In Fig.
5, we plotted the encoder output and the attention weights, as well as the word positions in the audio.
One hypothesis for an important function of the encoder is the detection of frames which should not be attended on by the decoder, e.g. which are silent or non-speech.
Such a pattern can be observed in Fig.
By performing a dimensionality reduction (PCA) on the encoder output, we can identify the most important distinct information, which we identify as silence detection and encoder time position , compare Fig.
Similar behavior was shown by Palaskar and Metze [2018].
We further try to identify individual cells in the LSTM which encodes the positional information.
By qualitatively inspecting the different neurons activations, we have identiﬁed multiple neurons which perform the hypothesized function as shown in Fig. 4.
E n c od e r d i m e n s i on s y
ea h i ’ d n e v e r t hough t a bou t t h a t w a y i don ’ t r ea ll y l ook [ L
AUGH
TE R ] i gu e
ss
i ough t t
o s o m e ti m e s < / s > − 0 .0 .0 .
[Figure]
We also observed that the attention weights are always very local in the encoder frames, and often focus mostly on a single encoder frame, compare Fig.
The sharp behavior in the converged attention weight distribution has been observed before [Chan et al., 2015, Beck et al., 2018b,
Palaskar and Metze, 2018].
We conclude that the information about the label also needs to be well-localized in the encoder output.
To support this observation, we performed experiments where we explicitly allowed only a local ﬁxed-size window of non-zero attention weights around the arg max of the attention energies, to understand how much we can restrict the local context.
[Table]
The results can be seen in Table 9.
This conﬁrms the hypothesis that the information is localized in the encoder.
We explain the gap in performance with decoder frames where the model is unsure to attend, and where a global attention helps the decoder to gather information from multiple frames at once.
We observed that in such case, there is sometimes some relatively large attention weight on the very ﬁrst and/or very last frame.
Conclusion
We provided an overview of our recent attention models results on Switchboard, LibriSpeech and
WSJ.
We performed an analysis on the beam search errors.
By our improved pretraining scheme, we improved our Switchboard baseline by over 8% relative in WER.
We pointed out the high training variance of attention models compared to hybrid HMM/NN models.
We analyzed the encoder output and identiﬁed the representation of the relative input position, both clearly visible in the PCA reduction of the encoder but even represented by individual neurons.
Also we found indications that the encoder marks frames which can be skipped by decoder, which correlate to silence.
Acknowledgments
This work has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 694537, project "SEQCLAS") and from a Google Focused Award.
The work reﬂects only the authors’ views and none of the funding parties is responsibleforanyusethatmaybemadeoftheinformationitcontains.