From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Reject

From: NIPS.cc/2018/Workshop/IRASL/Paper19/Authors
title: Response to Reviewer #1
comment: Thank you for your review. We will improve the paper by reflecting the comments.

Below are the responses to your comments.


1. Interesting results, but the approach is not very novel. It is basically just stacking two state-of-the-art models and showing that results are better than any model by itself (which is obviously expected, since we now have a lot of extra parameters compared to using a single model alone). 

=> We agree that the cascade method itself may seem simple. However, we have identified weaknesses in each model and we think the cascade method is one of the good ways to improve it. As mentioned in the paper, the current method is only a baseline of our hybrid framework and contains many points to be improved and optimized. We believe these points can be discussed further as the future research direction. 


2. I find the training strategy shuffling the order of the models interesting, but how practical is it? Speaking about practicality, this model requires a 500 ms window in order to enhance a signal, which makes it impractical for real-time applications.

=> We agree that the processing of the baseline models we used may not be immediately applicable to real-time applications. However, please note that these constraints are in each model rather than in the hybrid strategy. (Also note that, MDPhD 3M + 3M takes about 20ms process time per 1s with Tesla P40 GPU, so processing latency is more important than processing time.)
In addition, the mentioned 500ms means processing latency. This hop-size of 500 ms is fixed in the training process, but we can reduce this latency for inference by using smaller window hop-size.


3. I don't see the usefulness of the tests with sinusoidal noise, as very long high-frequency tones are clearly not natural in speech and most speech enhancement systems will be able to get rid of such tones.

=> We reported the results because it was most effective in identifying the limitation of current end-to-end approach which usually does not consider any additional modules. We also think that high-frequency is one of the possible noise types from the machine.


4. I find that the speech quality metrics used (Csig, Cbak, Covl and especially PESQ) are not very good metrics for speech enhancement in general, and would like to have seen some listening tests as well (even if with a small quantity of listeners).  

=> Thank you for your suggestion. We totally agree with you. We also recognize the problems of these quality metrics and know that they are sometimes uncorrelated with cognitive quality. Reflecting your comment, we plan a MOS or preference test to make up for this. Until this, please refer our page ( https://mdphdnet.github.io ) where readers could directly hear and compare our model with the other existing models. 


5. Also, why are so many values missing from the objective quality table? It would be nice to explain so the reader doesn't have to wonder why that is the case.

=> Thank you for your comment on what we have overlooked. We excerpted those indicators from their papers. Thus, the indicators that are not reported in the original paper could not be identified. We will clarify this fact in the paper.

From: NIPS.cc/2018/Workshop/IRASL/Paper19/Authors
title: Yes. The results are from the same dataset split.
comment: Yes. All of the compared models used the same training and test sets. You can check more detailed information and download dataset here ( https://datashare.is.ed.ac.uk/handle/10283/1942 ). We will clarify this in the main manuscript.

From: NIPS.cc/2018/Workshop/IRASL/Paper19/AnonReviewer1
title: Absence of PESQ and SSNR
comment: Regarding 4., are those results for exactly the same dataset split?

From: NIPS.cc/2018/Workshop/IRASL/Paper19/AnonReviewer1
title: Not very novel method, results are unclear
review: Interesting results, but the approach is not very novel. It is basically just stacking two state-of-the-art models and showing that results are better than any model by itself (which is obviously expected, since we now have a lot of extra parameters compared to using a single model alone). I find the training strategy shuffling the order of the models interesting, but how practical is it? Speaking about practicality, this model requires a 500 ms window in order to enhance a signal, which makes it impractical for real-time applications.

I don't see the usefulness of the tests with sinusoidal noise, as very long high-frequency tones are clearly not natural in speech and most speech enhancement systems will be able to get rid of such tones.

I find that the speech quality metrics used (Csig, Cbak, Covl and especially PESQ) are not very good metrics for speech enhancement in general, and would like to have seen some listening tests as well (even if with a small quantity of listeners).  Also, why are so many values missing from the objective quality table? It would be nice to explain so the reader doesn't have to wonder why that is the case.
rating: 3: Marginally above acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper19/Authors
title: Response to Reviewer #3
comment: Thank you for your constructive and detailed comments. We will improve the deficiencies of the paper by reflecting your comments. 

Below are the responses to your comments for clarification.

1. The paper is well written, some details are missing (e.g. which window was used in computing of FFT, which is relevant for the decision of the analysis overlap, how was the TAS-net method "reduced", ... )

=> We used Hanning window in this paper (We will clarify it). The FFT window size (512) and hop size (256) are described in section 3.1. 
For Tas-Net, we reduced the total layer depth and the channel size of some layers to control the parameter size. Furthermore, we replaced the convolutional block consisting of dilated group convolution operation followed by 1x1 convolution operation with dilated convolutional block (We described the detailed configuration in Appendix. We will explicitly refer this point in the main manuscript). In our interpretation, Tas-Net works well due to its auto-encoder structure and stacked dilated convolution layers, and in fact, our simplified model which follows this structure showed competitive speech enhancement performance.


2. I am not convinced about the significance of the reported results. Results show that TAS-net is better than the proposed combination in babble noise while the U-net is better in high-frequency steady noise. However, the proposed combined method is better when both the babble noise and the high-frequency noise are present. I do not find this result surprising. 

=> What we wanted to confirm in table 1 is that each single model shows poor performance for a particular noise type. This issue is likely to be severe when the models are applied to real-world speech enhancements where the noise patterns are diverse and unknown. Most deep learning-based speech enhancement studies have performed entire analysis on test set of various environments without considering or analyzing a specific noise type.
We attempted to demonstrate that the denoising performance may differ according to the time-resolution trade-off, and we think that this issue is of interest to the speech enhancement community. We also believe that researchers working on end-to-end deep learning models should be interested in the fact that Tas-Net, which has similar performance to U-Net, is very weak to the certain type of noise, e.g., high-frequency steady noise.

In addition, we want to add some comments on table 1. For example, the limitation of D (3M, Tas-Net) for high-frequency is not solved by increasing the model size or modifying the loss objective due to its inherent nature (table 1, Appendix A). This problem can be solved by our proposed method. Although our model does not outperform U (3M) for high-frequency noise, it shows comparable performance. Conversely, our model shows the performance similar to D (3M) even for babble noise and remarkably outperforms U (3M).


3. Table 2 shows the results of evaluations with existing techniques, unfortunately the two techniques used in the combination the TAS-net and the U-net are not compared. 

=> Thank you for your important comments on what we have overlooked. We agree that informing the performance of both models would be helpful for readers to better understand our hypothesis. The performance on the test set is as follows.
		               CSIG	CBAK	CVOL	 PESQ	SSNR
D (3M)    	         3.80	  3.29	  3.18	   2.57	  9.65
U (3M)     	         3.65	  3.21	  3.05	   2.48	  9.34
MDPhD (3M+3M)  3.85	  3.39	  3.27	   2.70	  10.22

We will add these results to the paper.
To summarize, we empirically showed that 1) two approaches have complementary denoising ability  2) our proposed method utilizes this complementary property 3) our hybrid approach can improve overall performance compared to each single model.


4. I am not clear why the PESQ and SSNR are not shown for the Wavenet and the MMSE-GAN techniques.

=> We excerpted those indicators from their papers. Thus, the indicators that are not reported in the original paper could not be identified. We will clarify this fact in the paper.

From: NIPS.cc/2018/Workshop/IRASL/Paper19/AnonReviewer3
title: Well written but not clear what is the contribution - looks like an ad hoc combination of two existing techniques U-net and TAS-net The evaluation not entirely convincing. 
review: The paper deals with enhancement of noisy speech. It combines two existing techniques  (TAS-net and U-net) and applied them in series in both possible combinations,  U-net followed by TAS-net and  TAS-net followed by U-net. Results of both combinations  are combined by summation. 

Clarity

The paper is well written, some details are missing (e.g. which window was used in computing of FFT, which is relevant for the decision of the analysis overlap, how was the TAS-net method "reduced", ... )

Originality

The combination of time-domain and time-frequency domain processing is original.

Significance

I am not convinced about the significance of the reported results. Results show that TAS-net is better than the proposed combination in babble noise while the U-net is better in high-frequency steady noise. However, the proposed combined method is better when both the babble noise and the high-frequency noise are present. I do not find this result surprising. Table 2 shows results of evaluations with existing techniques, unfortunately the two techniques used in the combination the TAS-net and the U-net are not compared. I am not clear why the PESQ and SSNR are not shown for the Wavenet and the MMSE-GAN techniques.

Pros

Clear writing.
The combination of the two techniques in the  cascade,

Cons

Results are not fully convincing. 
rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct