From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper25/Authors
title: Evaluation of Google ASR and model prototype
comment: The authors would like to thank the reviewer for the feedback.

1) Evaluation of Google ASR: We did not retrain the Google ASR. We used the Google cloud speech-to-text API, operating in video mode. It is available at: https://cloud.google.com/speech-to-text/

Evaluation:
> Generator from the CycleGAN operates on the spectral and aperiodic features extracted from the input utterance.
> The laughter-speech to normal speech transformed features (output of CycleGAN generator) are then used to synthesize the spoken utterance using WORLD Vocoder. 
> The synthesized utterance and the original utterance were used as inputs to the Google cloud speech-to-text  API.
> The text output obtained from Google speech-to-text is compared with the manual annotations of the utterances to compute word error rate (WER) and sentence error rate (SER).

2) Please find the overview of our model at :  https://drive.google.com/open?id=1cPtSjZFld2vxNXx3Z09epMFK2X5LkVrK
We are planning to share the working demo after anonymity period.

From: NIPS.cc/2018/Workshop/IRASL/Paper25/Authors
title: Anonymous link to the model prototype
comment: Dear Reviewers,

Thank you very much for providing your valuable comments and suggestions. 

We are planning to share a fully working demo of our model along with code after the review period. To maintain anonymity, we are providing a prototype of our model demo in the link: https://drive.google.com/open?id=1cPtSjZFld2vxNXx3Z09epMFK2X5LkVrK

Request you to check the prototype and provide your valuable feedback.

From: NIPS.cc/2018/Workshop/IRASL/Paper25/Authors
title: Authors response to the Reviewer's feedback
comment: The authors would like to thank the reviewer for providing detailed review and suggestions for further improvement.

1) We have not tried training an ASR on emotional speech alone mainly due to lack of enough training data. However, we would like to try the suggestions provided by the reviewer.

2) We did not observe any degradation in ASR performance on normal speech  when it was passed through the proposed front-end. We hypothesize that the use of identity loss function during the training of our model helps to preserve the characteristics of the normal speech.

3) The authors agree with the reviewers comment that "the models are emotion-class and gender specific, which would mean that before using the front-end speech will have to classified into these categories". We are working on building a gender and emotion agnostic system to handle different emotions and genders. 

4) We will include the block diagram of the model as an appendix to the final version of the paper.

5) We have used the aperiodic features which are well discussed in the literature for speech synthesis [Reference: https://pdfs.semanticscholar.org/1c42/4745996fd31920cadeec62a5514f3ce1a8b8.pdf?_ga=2.135015159.704166546.1542109585-2112611230.1537879943]

The laughter-speech regions contain higher amount of aspirations resulting in higher aperiodic component compared to that in normal speech. This motivated us to use the aperiodic components along with the spectral features. Similarly, the irregular excitations in creaky speech produce random components which result in stronger aperiodic components compared to normal speech.

We also observed that using only spectral features to transform laughter-speech to normal speech resulted in undesirable speech-like artifacts in the laughter dominating regions. Similar observation was also made for transformation of creaky speech. Adding aperiodic features significantly reduced these artifacts and resulted in better ASR performance.

6) F0 values are  not part of the feature set. They are used by the vocoder to synthesize better quality speech (for ASR).

7) We used the Google cloud speech-to-text API, operating in video mode. It is available at: https://cloud.google.com/speech-to-text/

8) The authors completely agree with the reviewer's comment "The analysis of the learned front-end is a little tricky to interpret.... It may not really represent emotion, unless the acoustic classes are already accounted for.". Since previous work has shown that laughter perturbations in speech affect the vowel regions, the visualization in Figure 1 was deliberately chosen to highlight the differences in the acoustic classes (or phonemes) for the laughter speech and the normal transformed speech.

9) Regarding the typos: Thanks for pointing out the typos. We will correct the typos in the final version of the paper.

From: NIPS.cc/2018/Workshop/IRASL/Paper25/AnonReviewer4
title: Interesting paper, but lacks enough experimental validation
review: The authors propose to use CycleGANs as a frontend for enhancing laughter and creaky speech to match normal speech. For this, they build gender- (male and female) and emotion-class-specific (laughter, creaky and normal in experiments) models using data from AMI and Buckeye corpus. As features, the authors show that using aperiodic components and mel filterbank based features work better than using the latter alone. Results show that the proposed frontend gives large improvements in word error rate when using off-the-shelf recognizers.

Overall, the paper is quite interesting and addresses an interesting form of degradation. It is hard to collect paired clean-degraded speech for emotional speech, and CycleGANs address this problem well. It is interesting to see that without using any paired data, the proposed frontend is able to get large improvements in performance. 

One important result that is missing in the paper is that it doesn’t show how well the ASR system would work if it were trained directly on emotional speech and normal speech. Typically, data addresses a majority of the problems with mismatch, making typical enhancement frontends redundant and unnecessary. Is this true for emotional speech as well? The ASR system can also be trained without using parallel data, and hence does not need any extra training data compared to CycleGAN based enhancement frontend. 

Another missing result is that the authors do not show how the frontend affects performance of normal speech. Moreover, the models are emotion-class and gender specific, which would mean that before using the frontend speech will have to classified into these categories. That task can introduce errors of its own -- something that is not addressed in the current work. It would be worth understanding how misclassifying emotion class affects performance. 

Minor
It would be useful to include a block diagram of the architecture used in the paper, to make it easier to understand the overall architecture and the losses used in the system.

Can you please add more details about aperiodic features  (APs) and include a citation? What do APs capture that mel filterbank does not? 

Why is it important to use log F0 as a feature? How much does it help improve performance?

Would be useful to include more details about the ASR system. For e.g., Google cloud API offers many modes of operation. Which mode did the authors use?

The analysis of the learned frontend is a little tricky to interpret. Note that clustering speech at a such a high level would most likely show the most salient differences, which in the case of speech is most likely going to be the acoustic classes (phonemes, e.g.). It may not really represent emotion, unless the acoustic classes are already accounted for.

Typos
Abstract: degrade the ASR performance -> degrade ASR performance

Intro: performing in par with the human -> performing at par with human

Intro: modeled by using -> modeled using

Intro: does not affect the ASR -> does not affect ASR

Sec 2: modeling the prosody-based -> modeling prosody-based

Sec 3.1: used the instance normalization -> used instance normalization

rating: 3: Marginally above acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper25/AnonReviewer1
title: Good topic and very nice idea, but missing lots of information about the cylceGAN used in this work. Can the author share more details to reproduce results?
review: The paper presents using cycleGAN as a FE. The idea is to enhance ASR with speech perturbation for creak and laughing voices.

> How did the authors evaluate Google ASR  with and without perturbation.
This is not clear to me. Did you retrain Google ASR system or what? This should be more clear.
>It will be interesting to see feature ablation study for APs, MFBs in this setup?
> Can you share the code used in this paper?
> Which implementation did you use for CycleGAN

Some references are strange; e.g. [7]- is this conference or Journal, where to find it?
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper25/AnonReviewer2
title: Interesting short paper for workshop. Although the results seem promising, there is no hope for reproduction of the presented research. 
review: The paper presents a CycleGAN based front-end, for the enhancement of speech recognition with naturally perturbed speech and under two cases; laughter and creaky perturbed speech. The evaluation of the proposed method uses some established datasets and ASR systems, and presents the results in terms of SER and WER. 

Cons: 
- Although that the paper claims that: "Analyze the performance of state-of-the-art ASR systems tested with naturally perturbed speech, including laughter and creaky speech.", the paper actually uses **only** laughter and creaky speech, not"including", but "only". The paper could benefit from toning down such claims. 

- The paper does not provide at all any means for reproducing the results. The description of the actual model is cryptic, not letting any hopes for reproducing from scratch the proposed method. This could be overlooked (since it is a short paper for a workshop) if there was some link to online code (with the actual repo omitted for review purposes). But there is not any link or an indication that will be a link to an online repository with the code. 

- The actual method seems not to be novel either, since the parts that are mentioned can be found at references [31, 33-35] (though, the paper **does provide** these references when presenting the method). 

----------------------------------------------

Pros:
- Interesting application
- Good increase of performance with the adoption of the proposed method


rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper25/AnonReviewer3
title: Cycle-Consistent GAN Front-end to Improve ASR Robustness to Perturbed Speech
review: The paper proposes a cycle-consistent adversarial
network (CycleGAN) for perturbed to normal speech conversion.
A 24-dimensional Mel Filter Bank (MFB) and 24-dimensional aperiodic components (APs)
are used as features for the Google cloud ASR and Kaldi ASR (with ASpIRE models).

It seems that the proposed CycleGAN with the same MFB has been proposed in a previous paper on voice conversion and, in any case, there is no evidence of significant novelties in the proposed neural architecture.

Word error rate (WER) and sentence error rate (SER) results are presented for laughter and creaky spoken conversions. 
Significant reductions are reported, as expected, by adding Aps to MFB features when Google cloud ASR is used and minor reductions are observed with Kaldi ASR. 
Nevertheless, SER is still very high for both ASR systems.


rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct