From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Reject

From: NIPS.cc/2018/Workshop/IRASL/Paper23/Authors
title: Authors' response on comments of AnonReviewer1
comment: The authors would like to thank the reviewer for the comprehensive reviews and suggestions. Below are authors' response to the review comments.

Comment : - the baseline systems are not really baseline systems as both are only sub-models of the proposed approach. To be able to compare the results, another approach should be added to strengthen the results, preferably another signal enhancement approach that also optimizes for speech-recognition performance

Authors’ response : The authors are keen on comparing the proposed approach with existing signal enhancement approach that also optimizes for speech-recognition performance. We kindly request you to suggest some implementations/references which can help us to achieve this. 



Comment : - Would it help to add the clean data to the train set? The authors claimed that the proposed method only need noisy data, however, it would be nice to see a direct comparison with a model, trained on a joint train set, which contains noisy and clean data.

Authors’ response : As mentioned in the paper (Section 3.1), the original training dataset was used as it is without using additional noises. However, the original utterances were recorded in a closed room scenario, and contain some amount of noise. Hence, no explicit clean data is available for the training. Our model learns only using these original utterances. We added external noise only in the test utterances for evaluating the robustness. As the reviewer might be aware, previous work on multi-condition training has shown that adding clean data in training phase did not give any significant improvement on overall performance [1]. We would be interested to try if it helps in our approach on a different dataset, where clean as well as noisy utterances are available.

[1] Y. Qian, M. Yin, Y. You and K. Yu, "Multi-task joint-learning of deep neural networks for robust speech recognition," 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Scottsdale, AZ, 2015, pp. 310-316.


Comment : A significance test, especially for Table 2. would make clear if 'masking enhancement' is really better in comparison to the baseline systems. 

Authors’ response : While we did not conduct an explicit significance test, we repeated our experiments with 3 different random initializations. We compared the average accuracies of T-F masking enhancement and direct enhancement performances  in case of noisy test set (Table 2). We found that the differences in performance still hold true for the average accuracies. T-F masking approach gives better results than direct enhancement, especially in the case of 10dB and 5dB noise. 


Comment : The approach is only tested for a command word classification. It is not clear how it would fare with a large vocabulary ASR, trained only noisy data only

Authors’ response : We are interested in extending this approach for the ASR task. Meanwhile, for analysis we passed a longer continuous speech utterance through the T-F masking enhancement block of our model (trained on speech commands). We had similar observations as we reported in Figure 2 of our paper. The enhancement block still tries to locate speech/non-speech boundaries and operates differently on speech/non-speech regions. While T-F regions containing the acoustic information are still preserved for the clean as well as noisy utterance, conclusions cannot be made on the performance on this utterance since the model is not trained for this task and vocabulary. This forms the motivation to explore current approach on ASR task. 

Comment : 

1) there are some typos, especially some articles are missing
2) In Figure 2-I, labels for the columns would help to understand the figure, even if the description is in the text. Also, it would be reasonable to select a b/w color scale, or at least a scale where very high and very low energy levels don't look the same in b/w printout

Authors’ response : Thank you for these comments. We will make sure to address these points in the final version.

Comment  : Why is there a version of only 20 classes in Table 1

Authors’ response : These results were added to compare with the most recent work (reference [14]) proposed on the speech command dataset, which reports accuracy on 20 classes.

From: NIPS.cc/2018/Workshop/IRASL/Paper23/Authors
title: CNN v/s LSTM, input pre-processing, and variations due to random initialization
comment: The authors appreciate the comments of the reviewer. Please find below our response to your comments.

Comment : The authors show how including the masking approach improves performance, but have not tested any other baselines besides a pure LSTM, which is a weak baseline. Additionally, in this paper they show surprisingly good results for that baseline compared to a CNN and a capsule network on ref. 14., but do not offer any explanation on why that could be the case. Is there anything different regarding e.g. the number of filters used, input pre-processing?

Authors’ response : We observed that CNN models fail to perform better than the LSTM baseline, without any data augmentation on this database. We hypothesize that this is due to lack of temporal shift invariance when convolution feature maps are collapsed and fed to fully connected layer. This problem is well handled by an LSTM model due to its sequence modelling capability. However, data augmentation, which introduces temporal shifts in input utterances during training, has been shown to improve the performance of CNN on this database. Since in [14] no such augmentation is used, we directly compare our results with them.  We used the same feature set and pre-processing techniques as detailed in [14], except we do not use deltas and delta-deltas of features.


Comment : Also, the performance improvement is really small and I wonder if training the models again with a different validation set or random seed would yield similar results. 

Authors’ response : Following the re viewer's comment, we repeated the experiments with 3 different random initializations. We then compared the average accuracies of T-F masking enhancement and direct enhancement performances  in case of noisy test set (Table 2). We found that the differences in performance still hold true for the average accuracies. T-F masking approach gives better results than direct enhancement, especially in the case of 10dB and 5dB noise conditions. 


Comment : There are several grammar issues in the paper, so it would be nice to give it a good review before the final submission.

Authors’ response : Thank you for these comments. We will make sure to address these points in the final version.

From: NIPS.cc/2018/Workshop/IRASL/Paper23/Authors
title: Response on state-of-the-art and extending to ASR task
comment: We would like to thank the reviewer for the comments.

While the proposed approach performs better than our baseline model, it cannot be concluded that it is state-of-the-art. Deeper and more complex architectures, data augmentation, and system combination techniques might produce better results. As opposed to achieving state-of-the-art results, the focus of our work is more towards developing an interpretable as well as robust network architecture.

We are keen on extending this approach for large vocabulary continuous speech recognition. Meanwhile, for analysis we passed a longer continuous speech utterance from Librispeech database through the T-F masking enhancement block of our model (trained on speech commands). We had similar observations as we reported in Figure 2 of our paper. The model still tries to locate speech/non-speech boundaries and operates differently on speech/non-speech regions. While T-F regions containing the acoustic information are still preserved for the clean as well as noisy utterance, conclusions cannot be made on the performance on this utterance since the model is not trained for this task and vocabulary. However, this forms the motivation to explore current approach on ASR task. We believe that this work is extendable to the ASR task.

From: NIPS.cc/2018/Workshop/IRASL/Paper23/AnonReviewer3
title: Good results, but fairly simple idea and limited experiments
review: The paper presents a simple idea of a CNN and LSTM combined network to achieve end-to-end speech enhancement and classification. The extra advantage of the proposed approach is that it can be trained only on noisy data without requirements for a parallel corpus of clean and noisy utterances.

The enhancement model is a DNN that provides a T-F mask like formulation. It is trained to improve classification probability and provides a good visualization of the T-F points important for classification.

The presented results are good, but it should be clarified if these are state-of-the-art and if the gains are significant. Also, it would be interesting to see this method applied to a bigger task, i.e. ASR task, and see how and if it helps performance. 
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper23/AnonReviewer2
title: Interesting results, although not 100% convincing
review: The paper provides very interesting insights into interpretability of ASR robustness. Although training an enhancement system and a recognition system end-to-end is not a new approach, the novelty here is that enhancement is performed explicitly (as opposed to learning robust features), although without requiring clean features as a target.

The authors show how including the masking approach improves performance, but have not tested any other baselines besides a pure LSTM, which is a weak baseline. Additionally, in this paper they show surprisingly good results for that baseline compared to a CNN and a capsule network on ref. 14., but do not offer any explanation on why that could be the case. Is there anything different regarding e.g. the number of filters used, input preprocessing? Also, the performance improvement is really small and I wonder if training the models again with a different validation set or random seed would yield similar results. 

There are several grammar issues in the paper, so it would be nice to give it a good review before the final submission.
rating: 3: Marginally above acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper23/AnonReviewer1
title: Robust Speech Command Recognition using Label-Driven Time-Frequency Masking
review: Summary:
 - The paper describes a keyword classification for noisy conditions, trained on noisy only data. The classifier is trained in an end-to-end manner and learns a (T-F) mask for the enhancement of noisy input data. 

Positive:
 - the approach returns a really nice visualization of the enhancement
 - the paper is well written and easy to follow
 - the training does not require the clean data
 - the mask does not try to enhance the signal but to interpret the audio signal in such a way as to distinguish between relevant and non-relevant information

Negative:
 - the baseline systems are not really baseline systems as both are only sub-models of the proposed approach. To be able to compare the results, another approach should be added to strengthen the results, preferably another signal enhancement approach that also optimizes for speech-recognition performance
 - Would it help to add the clean data to the train set? The authors claimed that the proposed method only need noisy data, however, it would be nice to see a direct comparison with a model, trained on a joint train set, which contains noisy and clean data.
 - A significance test, especially for Table 2. would make clear if 'masking enhancement' is really better in comparison to the baseline systems.
 - The approach is only tested for a command word classification. It is not clear how it would fare with a large vocabulary ASR, trained only noisy data only
- please cite: End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (Zhong-Qiu Wang, Jonathan Le Roux, DeLiang Wang, John R. Hershey)

 Comments:
  - there are some typos, especially some articles are missing
  - Why is there a version of only 20 classes in Table 1?
  - In Figure 2-I, labels for the columns would help to understand the figure, even if the description is in the text. Also, it would be reasonable to select a b/w color scale, or at least a scale where very high and very low energy levels don't look the same in b/w printout 

rating: 3: Marginally above acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature