From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept (Demo)

From: NIPS.cc/2018/Workshop/IRASL/Paper69/AnonReviewer2
title: Samples
comment: Thanks for posting the samples. Just a note, I couldn't play them directly from the browser, but downloading directly from the links (via wget, in my case) seems to work. Could just be my browser + OS combo (Chrome 67.0.3396.99, Ubuntu 16.04).

For the spectrogram, I was mostly referring to the lack of harmonics and texture especially in the upper frequencies - compare to even the right half of  Figure 2 for the reference you posted (Jia et. al.), which is still somewhat lower than the quality I am used to. It may well be a visualization or scaling issue, or could simply be the impact of noisy training. Unsurprisingly, WaveRNN seems to be able to "fix" this issue regardless so it shouldn't be a huge impact on the full system.

Thank you for posting samples, they sound quite promising.

From: NIPS.cc/2018/Workshop/IRASL/Paper69/Authors
title: Response to AnonReviewer4
comment: We thank the reviewer for the great feedback. Below are the itemized responses regarding each comment. We will also incorporate these into the revised version.

Re: it would've been very illustrative to also build a "second baseline" system that uses no additional encoding from z_s or z_r but the original clean data prior to the added-noise simulation. Such a system would've provided a top-line "oracle" performance for MOS, and allow us to see where in that baseline/top-line range the proposed approach falls.
Ans: We thank the reviewer for the great suggestion, and we are now training the baseline model on clean VCTK data. We will update the results once it is available.
In addition, we have obtained the new MOS results that use a WaveRNN for synthesizing waveforms from the spectrograms (Table 4). The MOS score is 3.22 for the baseline, 4.52 for the proposed model conditioning on clean z_r, and 1.32 for the proposed model conditioning on noisy z_r. Note that the WaveRNN MOS score for the proposed model conditioning on noisy z_r is lower than the original Griffin-Lim MOS score, because the WaveRNN model trained only on clean speech was unable to synthesize speech well when the spectrogram contains loud background noise.
Jia et al. (2018) reports the “top-line” MOS score of a speaker embedding-based model trained on clean VCTK data, but the result is not directly comparable to this paper due to the difference in vocoder (i.e. that work used WaveNet (van den Oord et al, 2016)). We do not expect there to be a large difference in MOS rating between WaveNet and WaveRNN (as in this paper) vocoders; however, we have not had a chance to run direct comparison.
 
Re: Consider providing more details on lines 81-85. Lots of background work packed here. 
Ans: We thank the reviewer for the comments, and will improve that in the camera-ready version.

Reference:
Jia, Ye, et al. "Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis." arXiv preprint arXiv:1806.04558 (2018).
van den Oord, Aäron, et al. "WaveNet: A generative model for raw audio." SSW. 2016.

From: NIPS.cc/2018/Workshop/IRASL/Paper69/Authors
title: Response to AnonReviewer3
comment: We thank the reviewer for the great feedback. Below are the itemized responses regarding each comment. We will also incorporate these into the revised version.

Re: model is very very close to GM-VAE
Ans: We attempted to explain the difference between this work and GM-VAE clearly in Section 3, as “while GM-VAE learns to disentangle speaker attributes from the rest by encoding those with small within-speaker variance.” The synthesizer and the two encoders are indeed of the same neural network architecture as in GM-VAE; however, this work focuses on disentangling highly correlated speaker and acoustic condition, and the novel features of this work are data augmentation and augmentation adversarial training for disentanglement, which are entirely different from GM-VAE. Furthermore, these techniques are complementary to GM-VAE, and be integrated to further improve the disentanglement performance of GM-VAE. We believe these features distinguishes our work from GM-VAE.

Re: synthesized samples supporting the claims would have made it very convincing
Ans: We created a demo page for the paper. Audio samples can be found here: https://google.github.io/tacotron/publications/adv_tts/
We hope the audio samples would make the claims more convincing.

Re: It is not clear how the model will be used to do clean TTS for same noise present in each sentence of the speaker.
Ans: We thank the reviewer for the great suggestions, and indeed using the same class of background noise (e.g. fan noise) may be more challenging than the scenario we considered in this paper. We will investigate the ability of our proposed approach to address such a scenario in the future work. However, we would also like to emphasize that the scenario considered in this paper is also a reasonable setting, as it is still easier to collect speech of the target speakers recorded in various noisy conditions, than to collect clean speech for all the target speakers record in carefully controlled environments. Our approach did show promising results dealing with such a scenario using real recordings from LibriSpeech.  Unfortunately we were unable to complete these experiments in time for submission to this workshop.

From: NIPS.cc/2018/Workshop/IRASL/Paper69/Authors
title: Response to AnonReviewer2
comment: We thank the reviewer for the great feedback. Below are the itemized responses regarding each comment. We will also incorporate these into the revised version.

Re: What does the model sound like trained on clean VCTK data? 
Ans: We have started training the baseline model on clean VCTK data, and will add the results when it is ready.

Re: Additive noise from CHiME data is a decent stand-in for in-the-wild data, but seeing the results on truly in the wild data as well would greatly improve the paper.
Ans: We thank reviewer for the great suggestions. This paper presents an initial proof of concept. Creating such a real speech-in-the-wild dataset and evaluating the proposed method is something we will definitely pursue in the future work.

Re: Maybe this can embedding be used to infer embeddings for unseen speakers and do classification in few or one-shot settings?
Ans: This is also a great suggestions and we would like to investigate this in the future work. We think the number of speakers in VCTK is rather small (109), and such transfer learning from TTS to speaker verification would perform much better if we train the proposed TTS model on a larger scale dataset as shown in Jia et al 2018.

Re: The resulting spectrograms look fairly poor, which is somewhat surprising to me. This could be a visualization / scaling issue, but without audio I cannot tell. Are there audio samples available? 
Ans: We created a demo page for the paper. Audio samples can be found here: https://google.github.io/tacotron/publications/adv_tts/ 
We are not sure if the reviewer is referring to all three spectrograms or just one of those, and which aspects of the spectrogram(s) the reviewer thinks is of low quality. If would be very helpful if the reviewer can clarify this comment, so we can better address the specific concern.
We did configure the colormap to have a smaller dynamic range by setting a smaller maximum value. Time-frequency bins with energy that exceeds the maximum value of the colormap are thus plotted with the same color (light yellow). This is for increasing the contrast of different energy values within the dynamic range, in order to make the background noise more visually perceptible in the spectrogram plots.


Reference:
Jia, Ye, et al. "Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis." arXiv preprint arXiv:1806.04558 (2018).

From: NIPS.cc/2018/Workshop/IRASL/Paper69/Authors
title: Audio samples available on the demo page
comment: We thank all the reviewers for their thoughtful comments and great suggestions. We have created a demo page to present audio samples: https://google.github.io/tacotron/publications/adv_tts/

We will address each reviewer's comments in the corresponding thread.

From: NIPS.cc/2018/Workshop/IRASL/Paper69/AnonReviewer2
title: Really nice writeup, but needs audio samples
review: What does the model sound like trained on clean VCTK data (with speaker embeddings, perhaps on a subset of all speakers if diversity is too high) directly? This is the exact setting from char2wav (Sotelo et. al.), so it could be a useful "ideal" benchmark. Additive noise from CHiME data is a decent stand-in for in-the-wild data, but seeing the results (no matter how poor) on truly in the wild data (sports announcers, public speeches, etc.) as well would greatly improve the paper - real world noise settings are generally quite different from CHiME, and might even be more amenable to synthesis.

Additionally, there is some potential to apply this architecture almost directly for multi-speaker TTS, using two embeddings, one adversarially trained to minimize speaker information, and one direct embedding. This isn't the setting the authors explore, but given the close similarity to their architecture it could be an interesting extension.

The plots of embeddings seem useful, and given the separability of the embeddings under t-SNE it is not too surprising the speaker classifier does well. Maybe this can embedding be used to infer embeddings for unseen speakers and do classification in few or one-shot settings?

The resulting spectrograms look fairly poor, which is somewhat surprising to me. This could be a visualization / scaling issue, but without audio I cannot tell. Are there audio samples available? 

This paper is well-written, and presents its methodology and results (barring audio samples) extremely clearly. Without audio samples (no matter the quality) linked, I can't fully recommend the paper acceptance but with that addition I would definitely be pro-acceptance.

EDIT: With the added audio samples, I improve my score significantly. The samples thoroughly demonstrate what the authors describe in the paper, and the full system quality is pretty good. Combined with the detailed writing, this is a clear accept in my mind. 
rating: 4: Top 50% of accepted papers, clear accept
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper69/AnonReviewer3
title: Interesting problem, no empirical analysis
review: The paper proposes a way to disentangle speaker from correlated noise.

It is not clear how this method can be used to synthesize clean speech, for speakers for which same kind of noise is present in every utterance. Ideally, same/very similar noise should be used for all utterances for the same speaker (in order to match the target use case)

pros
-- good visualization and numerical evaluation

cons
-- model is very very close to GM-VAE
-- synthesized samples supporting the claims would have made it very convincing
-- It is not clear how the model will be used to do clean TTS for same noise present in each sentence of the speaker.
rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper69/AnonReviewer4
title: Disentangling speaker identity from recording conditions for improving TTS quality in a multi-speaker setting
review: This paper presents an approach to disentangle speaker-identity from correlated acoustic-condition factors in a multi-speaker speech database to improve the quality of a text-to-speech system. The adopted baseline system consists of the Tacotron pipeline trained from phonetic inputs to produce Mel spectrograms (from which output speech is obtained via Griffin-Lim). This baseline is contrasted with one that is conditioned on  speaker and residual (acoustic) latent factors. Extracting these latent representations exploits two key ideas: a data augmentation approach where noisy replicas of the corpus speech that doe not correlate with speaker ID are created, and an adversarial training approach to encourage disentanglement between the acoustic and speaker representations.  

The paper is generally clear and very well written. It contains extensive validation of the approach, quantitatively and qualitatively in terms of crowd-sourced. mean-opinion scores. It is a significant and novel contribution since it addresses both a need for building TTS systems using found-data, and a shortcoming that pooling disparate resources will bring into the quality of the system (i.e., variable acoustic conditions). 


Regarding the evaluation in Section 4.4.3 -- I realize the point of this paper is to addresses the scenario where you wish to build a TTS system out of "found data" where you have no control over a variety of acoustic conditions. But it would've been very illustrative to also build a "second baseline"  system that uses no additional encoding from z_s or z_r but the original clean data prior to  the added-noise simulation. (In writing this, I'm assuming that the baseline described in this section corresponds to the standard Tacotron pipeline shown at the bottom of Fig. 1, trained with the artificially noisy data described in the procedure in lines 124-127.) Such a system would've provided a top-line "oracle" performance for MOS, and allow us to see  where in that baseline/top-line range the proposed approach falls.
 

Some minor comments:

-- Consider providing more details on lines 81-85. Lots of background work packed here. 

rating: 5: Top 15% of accepted papers, strong accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct