From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper30/AnonReviewer1
title: Interesting; probably accept
review: I only skimmed it but it seems to provide some potentially interesting analysis, e.g. about beams etc., of attention models.  I'd lean toward accept.
rating: 3: Marginally above acceptance threshold
confidence: 1: The reviewer's evaluation is an educated guess

From: NIPS.cc/2018/Workshop/IRASL/Paper30/Authors
title: Review comment
comment: Thank you for your review.

We have now also added results for our best model using a language model.

From: NIPS.cc/2018/Workshop/IRASL/Paper30/Authors
title: Review comments
comment: Thank you for your detailed review, it is greatly appreciated.
We have adjusted the paper slightly, in addition to the comments below.
Our goal for this paper is to understand attention-based models better
and use this knowledge to further improve our model.
That is why we have included extensive experiments on pretraining variants,
as we have experienced rather large performance improvements and
we wanted to understand this aspect better.
Additionally, we are not aware of any comprehensive investigation on
pretraining for attention-based models, thus we wanted to share our results
which arguably provide discussion points.


> 1- Related works: Distinguish your work from [1].
Thanks, we supplemented the paper with the appropriate mention and short discussion.


> 2- Related works: [...] It would be more fitting if you would mention and summarize NLP works, specially the ones related to attention.
This would be indeed more appropriate, we have included relevant literature in the latest paper update.


> 3- multi-GPU training?
We used single-GPU training for all experiments.


> 4- In Table 1, what is the difference between "attention-this work1-none-BPE 1k" and "attention-[Zeyer et al., 2018b]-none-BPE 1k"?
this work^1 is the same exact baseline, though we have run it multiple times to select the best model.


> 5-In Table 1, "this work ^ 2" does not have a language Model but still is better than others which is interesting. [...]
Thank you, we agree that results for a combination with a language model would fitting.
For the initial paper submission we did not include this comparison, due to time constraints.
The relevant information has been added to the paper.


> 6- Pre-training experiments:  Interestingly enough best result is for 4 (and not 6) layers of LSTM with no pre-training. Could it be because of the pre-training scheme?
We suspect so too. For Table 4 the same pretraining scheme was used which is probably not optimal for the different architectures.
Even though we could not optimize each encoder configuration individually (due to computation/time constraints),
Table 6 and 7 show modifications to the pretraining scheme which resulted in further improvements for similar encoder configurations.


> 7- Line 124: “This might be due to more stable hyper parameters.” At this point, it is still not clear what is the difference in hyper-parameters.
Relevant information has been added the paper, we meant the BPE vocabulary.



> 8- Line 132: “It seems that pretraining allows to train deeper model and helps for the convergence in that case…” it is not obvious to from your experiments. Table 4 shows the opposite to some extent. [...]
We noticed a trend where pretraining helps large and deep models to converge without requiring to fine-tune the learning rate,
or other hyper parameters.
Also, models with an even deeper encoder did not converged without pretraining.


> 9- I suggest adding one row to show the baseline for all the tables.
Due to space constraints we choose to omit baseline results which we have shown earlier.


> 10- Author(s) mention an 8% relative boost in performance from their analysis. It would be interesting to compare with Random Hyperparameter Search preferably with the same number of experiments for a fair comparison.
We agree that this experiment would be interesting.
Unfortunately, we do not have the computational capabilities to perform an exhaustive hyperparameter search.


> 11-Line 168: “Related to that, we observe a high training variance. I.e. with the same configuration but different random seeds, we get some variance in the final WER performance.”
  This is an important information that is missing from all of the results table!
The random seed is fixed for all experiments, unless otherwise noted.


> 12-Section 6 and Table 8: For a fixed random seed, where does the non-determinism come from? [...]
We suspect this variance stems from the non-determinism on various GPUs.
In particular, the tf.reduce_sum operation in tensorflow seems to be non-deterministic
as it uses CUDA atomics.
We have added a short discussion about this topic and included a link to further relevant information.


> 13-To interpret figure 4, one needs to see the histogram of length of the sequences. I am not sure averaging output units over the whole dataset would mean much if the distribution is heavily skewed. [...]
To allow comparison within the dataset, we have length-normalized all sequences.
The purpose of Figure 4 is to demonstrate that individual neurons have learned to represent the relative sequence offset.
Unfortunately, we have not found any directly-applicable literature.


> 14-Section 7 - Analysis of the encoder output : It is not clear how the authors found these single neurons.
We simply visually inspected the encoder activations by looking at figures similar to Figure 5.
Therefore, we could single out individual neurons which looked particularly interesting.

From: NIPS.cc/2018/Workshop/IRASL/Paper30/AnonReviewer3
title: Official Review: "A comprehensive analysis on attention models"
review: This work experimentally compares different hyper-parameters of a seq2seq model for Automatic Speech Recognition (ASR) with attention mechanism and performs some analysis on the trained models.
The title is not only too broad as it considers attention models only in ASR but also the analysis is not comprehensive enough. Among others, it looks at the errors of the beam-search, the effect of beam size, effect of number of LSTM layers in encoder, number of hidden units and the effect of pre-training. It would be much appreciated if you focused more on deeper analysis of the models rather than the structure of the networks, as you did in later sections.

The paper is easy to read and the introduction is well-written citing relevant works. However, going through the paper and only judging by the story-telling for the experiments, it seems that the author(s) are describing their effort at manually tuning a ASR model (which is not a bad thing by itself if presented right). If that is not what the author(s) intended to convey, I suggest re-writing and making new sub-sections for added clarity, specially for section 5.

Some points and questions:
1- Related works: Distinguish your work from [1].
2- Related works: You briefly mentioned analysis of models in vision tasks. It would be more fitting if you would mention and summarize NLP works, specially the ones related to attention.
3- RETURNN framework is for multi-GPU setting. Is this how you trained your networks?
4- In Table 1, what is the difference between "attention-this work1-none-BPE 1k" and "attention-[Zeyer et al., 2018b]-none-BPE 1k"? If their are the same, where does the performance gap come from?
5-In Table 1, "this work ^ 2" does not have a language Model but still is better than others which is interesting. It would be even so if we could see the same experiments with a language model. It is important because a high performing ASR system will most likely have one.
6- Pre-training experiments:  Interestingly enough best result is for 4 (and not 6) layers of LSTM with no pre-training. Could it be because of the pre-training scheme?
7- Line 124: “This might be due to more stable hyper parameters.” At this point, it is still not clear what is the difference in hyper-parameters.
8- Line 132: “It seems that pretraining allows to train deeper model and helps for the convergence in that case…” it is not obvious to from your experiments. Table 4 shows the opposite to some extent. If you are analyzing table 5, then this conclusion should be written be a bit later.
9- I suggest adding one row to show the baseline for all the tables.
10- Author(s) mention an 8% relative boost in performance from their analysis. It would be interesting to compare with Random Hyperparameter Search preferably with the same number of experiments for a fair comparison.
11-Line 168: “Related to that, we observe a high training variance. I.e. with the same configuration but different random seeds, we get some variance in the final WER performance.”
This is an important information that is missing from all of the results table!
12-Section 6 and Table 8: For a fixed random seed, where does the non-determinism come from? An expiation would make this clear.
13-To interpret figure 4, one needs to see the histogram of length of the sequences. I am not sure averaging output units over the whole dataset would mean much if the distribution is heavily skewed. If you know of a similar analysis in the literature, please cite them.
14-Section 7 - Analysis of the encoder output : It is not clear how the authors found these single neurons.

Prabhavalkar, Rohit, Tara N. Sainath, Bo Li, Kanishka Rao, and Navdeep Jaitly. “An analysis of “attention” in sequence-to-sequence models,“.” In Proc. of Interspeech. 2017.
rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper30/AnonReviewer2
title: A comprehensive analysis on attention models
review: The paper describes comparison of an attention based neural seq2seq architecture model with other architectures that do not use attention mechanisms. Experiments are conducted on various corpora 

Based on final word error rate (WER)  performance of different architectures it is shown that there are problems in the architecture using the considered attention based model without language model.

Results on error analysis and analysis on pre-training, training variance, and encoder output are also presented with interesting comments.

The paper does not propose any novel solution. It shows that a type of architecture with a type of attention mechanism has higher WER compared to other systems, some of which showing already low WER, without attention mechanisms.

The results have some interest perhaps suggesting that the for ASR other types of attention mechanism should be investigated 


rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct