Paper:
Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
We describe a neural network-based system for text-to-speech (TTS) synthesis that is able to generate speech audio in the voice of different speakers, including those unseen during training. Our system consists of three independently trained components: (1) a speaker encoder network, trained on a speaker verification task using an independent dataset of noisy speech without transcripts from thousands of speakers, to generate a fixed-dimensional embedding vector from only seconds of reference speech from a target speaker; (2) a sequence-to-sequence synthesis network based on Tacotron 2 that generates a mel spectrogram from text, conditioned on the speaker embedding; (3) an auto-regressive WaveNet-based vocoder network that converts the mel spectrogram into time domain waveform samples. We demonstrate that the proposed model is able to transfer the knowledge of speaker variability learned by the discriminatively-trained speaker encoder to the multispeaker TTS task, and is able to synthesize natural speech from speakers unseen during training. We quantify the importance of training the speaker encoder on a large and diverse speaker set in order to obtain the best generalization performance. Finally, we show that randomly sampled speaker embeddings can be used to synthesize speech in the voice of novel speakers dissimilar from those used in training, indicating that the model has learned a high quality speaker representation.

Review:
This paper discusses a voice cloning TTS system that is based on speaker embeddings as opposed to speaker adaptation. The speaker embeddings are found using a speaker encoder. The key idea of this paper is that the speaker encoder is trained separately on a speaker verification task instead of training it together with the TTS system. This reduces the amount of speech-transcript pairs required to train the complete system. The performance of the proposed system is extensively analysed and compared to the performance of a similar system using fixed embeddings for seen speakers, based on the speaker ID.  It is shown that the proposed system achieves similar performance for seen speakers, but is also able to generalise to unseen speakers, which is not possible for the fixed embedding system.  The quality of the paper is pretty good, but is lacking key baseline systems. The experiments section is extensive. Both quantitative and qualitative results are presented and several evaluation schemes are used. What I feel is lacking are additional baseline systems. The paper should definitely include a baseline where the speaker encoder is not trained on a speaker verification task, but is trained jointly, like in [1] or [8]. Otherwise we cannot assess the value of training the speaker encoder separately. If such a baseline is added we can also compare the results for unseen speakers. Adding a speaker adaptation system as a baseline could also strengthen the paper, but is not as crucial since a comparison has already been done in [1].  The author has made some good points in their response about the experimentation, so I would say that the argumentation is sufficient.   The clarity of the paper is good. The core concepts of the paper are explained well and references are extensively cited as comparison with related work. The components of the system are not explained in full detail but references are provided to papers that do explain the details. If the components and hyper parameters of the proposed model are the same as in the reference material the system should be reproducible.   I would say the originality and significance of the paper are mediocre. There is not a lot of novelty in the paper. Pre-training the speaker encoder has been done in [5], but a different type of speaker encoder is used. All the other components in the system are not novel. I think putting the components together as proposed in the paper makes sense, so in my opinion it is an interesting system for this task. I would say that it is likely that this system will be used for voice cloning TTS systems in the future.