From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper59/Authors
title: Author response
comment: Thanks for the review. Those are fair comments. Indeed, the study is preliminary and inconclusive. The results in our mind do offer some insights into the behavior of the various models explored, but there is much remaining to do (WER decoding in particular). 

Re: specific reviewer comments:

- The frame accuracies were computed on a subject of 1000 held-out utterances (with on average 120 frames/utt).
- The specific computation of frame accuracy (conditioned on ground-truth up to 1..t-1) is indeed just one of possibly many choices. The goal was to have _some_ sort of measure of frame accuracy to compare against baseline models. The measure we chose is admittedly arbitrary, and of debatable utility. WER results are really needed to shed light on this question.
- Good point, the reviewer noticed that the shallow features used were MFCCs, while the input to the nets for the deep features were logmels. We did not mention this in the current draft, but the logmels only produced very poor frame accuracies for the generative models. Presumably the use of MFCCs (and derivatives) gave better temporal context representation, as well as some measure of feature independence, alleviating the weakness of the radial density model.

From: NIPS.cc/2018/Workshop/IRASL/Paper59/Authors
title: Author response
comment: Thanks for the review. Those are all fair points. Some comments:

(1) PCA on the deep features: yes, that is the standard practice with tandem. In this study, with the radial densities, the only coping mechanism is the number of components/mixture. Certainly, PCA, or the use of full covariances, could make a significant difference. We have a number of results for full covariances, but did not include them in the current draft. (Prediction cost is much better with full covariances).

(2) Yes, there is a counter-intuitive result on the deep features / shallow densities -- the 32 components/mixture model has a slightly worse frame accuracy than the 10 components/mixture. In our minds, this may not be a significant difference. The log-likelihood results are as expected (having more components helps). Note that we are only reporting 2 significant digits for the frame accuracies.

(3) Thanks for the Grezl et al. and Heigold et al. Interspeech 2008 references. The latter in particular is very relevant to the discussion. Indeed, the equivalence is very clear for the radial model (esp. with a single component/mixture). Note that Heigold et al. did not consider SGD/ML training of the log-linear models, only SGD/CE training (as far as we can tell), whereas our study does explore SGD/ML.

(4) Smoothing: this is the TensorFlow TensorBoard moving average (https://stackoverflow.com/questions/42281844/what-is-the-mathematics-behind-the-smoothing-parameter-in-tensorboards-scalar), with smoothing factor of 0.95. We do not have an estimate of the variance of the results.  Note also that "convergence" was declared somewhat arbitrarily, and choosing different stopping points would affect the results reported. These metrics are admittedly not very rigorous, but we believe representative of the conditions explored. Some more details: 1000 held-out utterances (of an average of 120 frames/utt) were used to compute frame accuracies; these statistics are rather noisy (which led us to only report 2 significant digits). The costs were based on a running average on the entire training set (and are less noisy than the frame accuracies).

(5) Table 2, CD vs CI: Good point. We should have clarified that the CI and CD here are consistent for both features and density, i.e. both the deep features and the deep density are generated for either CI or CD phones, but we did not yet investigate the cross-combinations, which are indeed possible, as the reviewer suggests.

(6) I agree with the reviewer's point about "future blindness".

From: NIPS.cc/2018/Workshop/IRASL/Paper59/AnonReviewer3
title: Interesting connection between tandem and generative modeling, unclear evaluation
review: The authors present an interesting framework for generative acoustic modeling for ASR within the well-known HMM framework. However, the evaluation only provides frame error rates which in some cases even are counter-intuitive, when moving from single densities to multiple densities per phone for deep features with shallow density modeling.

The authors chose to not use PCA or similar postprocessing to decorrelate the deep features before using them in densities. Especially in view of the choice of using radial covariance this seems suboptimal, more so, as it is common practice in tandem modeling. This might have a measurable effect on the results obtained and corresponding conclusions.

Sec. 1.6: also relevant in this context would be the original work on bottleneck tandem:
Grezl et  al., Probabilistic  and  bottleneck  features  for LVCSR of meetings, ICASSP 2007.

Table 2: For the CE(-only) results, you might want to point out that the models even are equivalent, as shown in
Heigold et al., On the equivalence of Gaussian and log-linear HMMs, Interspeech 2008.

Sec. 6.2, last paragraph: pls. elaborate on how smoothing her is done and what would be the expected effect on the statistics obtained.

Table 2: what does CD/CI refer to: density, feature, or both? As the deep net's outputs are used as features, also a combination CI  deep feature with CD density model would be possible.

Summary/Sec. 7: to finally be able to conclude on the "future blindness" hypothesis, realignment/full recognition/WER results would be necessary, as training and evaluating within a given frame alignment possibly introduces too much constraint to the evaluation, and delaying decisions already might help.
rating: 3: Marginally above acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper59/AnonReviewer2
title: Interesting paper
review: The paper has a very nice introduction that motivates the author's search for a better way to combine models trained on different corpora than interpolation (or re-scoring). Some minor inconsistencies have already been pointed out below (e.g. S vs. W). I like the "deep density deep feature" interpretation of RNN-T leveraging Gaussian mixture densities, at least I haven't seen this anywhere else before - this justifies the paper's publication. On the other hand, the paper leaves the impression of being done in a hurry and not really complete: hardly conclusive performance indicators like frame accuracies computed on a "small held out dataset" (size?) and using the ground truth for 1..t-1. Experimental parameters are varied somewhat, but it is hard for the reader to understand why a certain choice was made and not another (just as a random example, deep feature models use a different feature vector than the shallow ones). Overall, interesting paper, and good for being presented and discussed at a workshop, but certainly not sufficient for someone who just reads the proceedings.
rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper59/Authors
title: Author response
comment: Thanks for the review.

There were definitely some unclear sentences in the Introduction, in particular relating to "the channel model". Let me try to clarify.
(1) Yes, Jelinek was the first to detail the model as it applies to speech recognition (reference is given). However, ...
(2) ... the real point of this part of the article is simply to highlight that ASR has traditionally used a principled approach to module combination, namely combining generative modules with Bayes' rule. As it stands, the discussion of channel model obscures that point since people may or may not be familiar with it. 
(3) Furthermore, the channel model can be applied to an end2end P(S|X), without necessarily using Bayes' rule, as the reviewer points out. Channel model != modular decomposition using Bayes' rule, the latter is consistent with the former, but not identical to it. So again, the Intro is misleading/confusing, guilty as charged.
(4) Some inconsistent notation (e.g. W and S should be merged into just S) makes matters worse.

So the point I was trying to make is just that ASR lends itself well to a modular decomposition p(X|S) * P(S),  using Bayes' rule, in contrast to the heuristic use of fusion with end2end models, e.g. P(S|X)^(alpha) * P(S)^(1-alpha). This work attempts to explore what a much better p(X|S) could be.

Concerning the other points of the review:
- The article describes quite a bit more than just "tandem features", it presents the 4 combinations of {shallow features, deep features} X {shallow density, deep density}. "Tandem features" would be deep features & shallow density, i.e. just one of the 4 models described here
- I think the results for the deep features are as expected, and the log-likelihoods more or less as expected across the board, but the frame accuracies for the shallow features are unexpectedly low -- especially for the auto-regressive models. Does the reviewer find the latter results to be predictable too? The question is relevant because the auto-regressive models are powerful models that do not require use of deep features (they are "pure" generative models). Perhaps frame accuracy here, and the way it was computed, is irrelevant and misleading. I'm very interested in feedback on that point.
- The reviewer is right, MMI doesn't imply a sequence-level cost function, the submission adopts ASR-speak. The phrasing can of course be fixed.

Re: fit to the IRASL workshop: in my mind this is a very good fit, as the goal of modular decomposition using Bayes' rule is much more _interpretable_ than e.g. the heuristic shallow fusion applied to end2end models. Furthermore, the 4th architecture is a twist on the well-known RNN Transducer, with a specific interpretation of a deep density net conditioned on labels predicting a deep encoding from acoustic features. Personally I find that to be an interesting interpretation / extension of RNN Transducer -- though clearly the results are preliminary and inconclusive.

From: NIPS.cc/2018/Workshop/IRASL/Paper59/AnonReviewer1
title: Tandem features on 20k hours
review: The paper describe a shallow and a deep way of computing features in an acoustic model, and then apply shallow and deep classifiers to these features, evaluating all combinations of systems.

The paper contains an interesting introduction, but is hard to reach; a native speaker needs to proofread it. I find many thoughts a bit confusing:

- are the authors claiming that Fred Jelinek was the first to apply the "noisy channel model" in speech recognition? I am not sure about that - if it is important, a reference would be good.
- i am also unclear about the meaning of the "channel model" - noise can be applied to both the acoustic model, and the language model, and so the noisy channel model can also be applied to a modern end-to-end model, which predicts a sequence of words, rather than a sequence of frames
- MMI is not necessarily a sequence training loss fnection - it can also en applied at the frame level
- also, it is not clear what aspect of the paper makes it a good candiadte for inclusion in the IRASL workshop
- etc

Finally, the results are somwhat unsurprsing, and shown on an unknown data-set, so that th reader cannot compare the proposed model to other models - this make the enire messave f the paper inclea
rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct