From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper27/AnonReviewer1
title: Basic and interesting experiments that might lead in the future to better understanding of how iterative inference and neural nets are learning masks and outputs for speech enhancement. It might also leads to the design of better models.
review: The authors propose incorporating Residual, Highway and Masking blocks inside a fully convolutional pipeline in order to understand how iterative inference of the output and the masking is performed in a speech enhancement task. The authors perform a qualitative analysis by observing the patterns and the learned frequency in a the spectrogram through the blocks.

Even though the work is at its preliminary stage, the idea of using residual, highway and masking blocks for interpretation leads to better understanding the process by which the iterative inference is helping neural network to estimate. Such basic and simple experiments might lead to the design of the appropriate masks, activations and losses that are required in order to obtain better performance. More advanced models like UNet with incorporated residual blocks (such in https://arxiv.org/abs/1608.04117 ) might also be beneficial.

rating: 4: Top 50% of accepted papers, clear accept
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper27/AnonReviewer2
title: An original analogy but shallow discussion.
review: The authors interpret highway, residual and masking connections. I appreciate the relation that is made with common approaches in speech enhancement (masking, spectral subtraction), and this is a real merit of the paper. It is an insightful analogy.
However, the paper is weakest on the aspect of "clarity". Line 99-101 describe two approaches, but I see no further discussion. Which architecture is reported on in section 3 ? Also the discussion is rather limited. What does this learn us about what is happening across layers and why highway/residual/masking behave differently ?

Some minor comments:
* section 2.1 line 91: reverberation times need a unit (seconds I assume)
* section 2.2 line 97: is standardization performed with new statistics estimated per utterance, or global statistics of the complete training set ?
* can you comment on the number of GRUs (256) relative to the number of frequency bands (257) ?
rating: 3: Marginally above acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper27/AnonReviewer3
title: Thank you for your clarifications
comment: >Our test set is composed of 1200 complete sentences, each 3-5 seconds long. We call a "sample" in this context a complete sentence, not a single frame. 

Thank you for clarifying this, I would avoid using the term "sample" and use "sentence" since it can be confusing in the context of STFT SE.

>We agree (and even state in the paper) that our results are not competitive; however, this is not our goal with this paper.

I agree that is not necessary to beat SoA for such and experiment. However, since neural networks are very complex, having sufficiently good performance compared to other methods reduces the chance that the shown phenomena are spurious  (i.e. under-fitting or under-trained models).

I will update my review on the experimental setup from "clearly not sufficient" to "not sufficient" since the other observations still hold (speaker dependent, custom synthetic noise corpus, weak results).  In my view, this does not justify increasing the rating.

From: NIPS.cc/2018/Workshop/IRASL/Paper27/Authors
title: Some clarifications
comment: The authors thank the reviewer for the very thorough review and would like to clarify some points.

1. The reviewer is absolutely right in stating our setup with log-magnitude is not similar to how spectral subtraction and masking happens in more traditional DSP methods. We thank the reviewer for bringing that up and will rerun our experiments using STFT-magnitude domain as inputs.

2. Our test set is composed of 1200 complete sentences, each 3-5 seconds long. We call a "sample" in this context a complete sentence, not a single frame. As our models use recurrent processing blocks, each sample is a sequence of all frames in a sentence. For minibatches, since sentences have different lengths, we only compute the MSE between valid frames for each sentence.

3. We agree (and even state in the paper) that our results are not competitive; however, this is not our goal with this paper. We intended to better understand how skip connections operate in a speech enhancement model and use those insights to point out that some of these models are more interpretable than e.g. stacked RNNs. We will try to state this goal in a clearer way in the paper.

4. We will be publishing our code as soon as our updated experiments using STFT-magnitude are complete. Our apologies for the delay in sharing it.

From: NIPS.cc/2018/Workshop/IRASL/Paper27/AnonReviewer3
title: Interesting idea but work is too preliminary
review: The paper explores an interesting idea but, as the authors indicate, this is preliminary work. The experimental setup is not sufficient. The comparisons between neural networks and traditional spectral subtraction and masking are also not fully convincing. Details follow.

# Experimental Setup not Sufficient

- The authors generate their own noisy speech by artificially adding noise from a well established noise data-set to a less know clean speech data-set. This is common practice but hurts comparability. Also using real noisy speech rather than synthetic one should strengthen the results. There are various corpora for SE/ASR matching the requirements e.g. Aurora4, CHiME and REVERB.

- [CORRECTED] The data-set created has noise and SNR range mismatch between train and test, which is good, but the data is single speaker. For a general claim like the one in the paper, this undesirable and weakens the paper. 

Some other problems with reproducibility/clarity

- Code to generate the noisy speech data-set should have been provided but is missing here https://anonymous.4open.science/repository/6487014f-35ad-4d1c-959e-5adc9316de95/

- The description of the data-set is missing the sampling frequency. I deduce it is 16KHz which gives 512 size window, from which the 257 coefficients under Niquist frequency are selected. This assumes the STFT is of the same size as the window. This does not need to be the case, despite being obvious, this could have been made more clear. The author(s) imply that they normalize the signals. I assumed this is in the time domain but this is again not clear.

Weak results

- The results are as well not good, as recognized by the authors, with only small improvements in acoustic quality for the different experiments and non competitive with state-of-the-art. At least a comparative baseline with SoA methods would be needed for reference.

# Comparison between NNs and traditional spectral subtraction / masking

Note: the author(s) describe masking layers. In this context I find Gated Linear Units (https://arxiv.org/pdf/1612.08083.pdf) relevant as they are exactly Masking networks for ratio masks, the most suitable of the Masking NNs (more on that below)

While it is true that residual/Masking/highway networks do work in a similar way as traditional Speech Enhancement (SE). SE in STFT/STFT-magnitude domains is justified by acoustic properties that do not hold in the Neural Network layers unless some provisions are taken. This provisions were not taken in this work, which makes the results less interesting:

- Spectral subtraction in the (complex) STFT domain exploits additivity (and therefore independence) of speech and noise signals. In the log STFT-magnitude domain, which is the input domain used in this work, this does not hold (see e.g. the ALGONQUIN model from Frey et al).

- Traditional spectral subtraction is applied in the magnitude domain. In this domain aditivity also does not hold either, due to the phase interaction between coefficients, but it may be accepted as an approximation. The exploration of additive models in log-amplitude domain is therefore not as meaningful as the amplitude or complex domains.

- The other property exploited: the natural sparsity of signal overlap in the STFT-magnitude domain i.e. each STFT coefficent is either speech or noise dominated would be maintained for 1-to-1 transformations, as it is the case here.

- the _masking_ network would fit into the sparsity model above if it used a sigmoid activations (GLU network). However, the authors indicate that they are using _linear_ activations. This introduces a non-linear component and I have difficulties fitting this into additive or sparsity models. It seems to me that this case should be avoided unless it presents some interesting properties.

- The _residual_ network does fit well into spectral subtraction, but if it had been applied in the STFT-magnitude and not the log-STFT-magnitude domain it would make more sense. 

In sum, if the models would have been set in a way that better exploits the natural properties of STFT domain, the paralelism between NNs and classic spectral subtraction / maskig would make more sense. A possible scenario in my opinion would be:

- STFT-magnitude domain as inputs (or maybe real and complex components of STFT domain)
- Residual, GLU  and Highway networks (log may be applied to the input of the mask/correction computing blocks)
rating: 2: Marginally below acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature