Paper:
Neural Voice Cloning with a Few Samples
Voice cloning is a highly desired feature for personalized speech interfaces. We introduce a neural voice cloning system that learns to synthesize a person's voice from only a few audio samples. We study two approaches: speaker adaptation and speaker encoding. Speaker adaptation is based on fine-tuning a multi-speaker generative model. Speaker encoding is based on training a separate model to directly infer a new speaker embedding, which will be applied to a multi-speaker generative model. In terms of naturalness of the speech and similarity to the original speaker, both approaches can achieve good performance, even with a few cloning audios.
2 While speaker adaptation can achieve slightly better naturalness and similarity, cloning time and required memory for the speaker encoding approach are significantly less, making it more favorable for low-resource deployment.

Review:
This work examines the problem of cloning a target speaker's voice given a trained generative model (with audio/speech and text). The broader context of the work is in Text to Speech (TTS) models in which rapid and excellent developments have occurred in the last few years. These models use DNNs as seq2seq models to predict speech to text. Other permutations are speech to speech/voice conversion (going from source voice to target voice), and voice cloning and adaptation, which this paper addresses. In the problem under consideration in the paper (voice cloning) one is able to produce voice that sounds like a certain 'target' or desired speaker using only a few samples presented which is then used to generate speech given arbitrary text. This generation is carried out using a multispeaker generative model that is trained over a large number of speakers and text/speech training data.   The reviewer feels that this work is an outstanding example of current TTS work and a massive undertaking both in terms of the quality of results produced and in the relevance of ideas it contains.  Prior work: The work is an extension of recent developments on seq2seq attention based TTS modeling works (e.g. Tacotron, DeepVoice-3 and lower). The models have several important improvements such as autoregressive models with dilated convolutions, trainable speaker embeddings, multiheaded self attention, and input feature processing (convolutions, filter banks etc from CBHG in tacotron), and using characters instead of phonemes to do away with segmentation.  The work uses the DeepVoice-3 architecture to carry out the voice cloning task. Cloning approach: Two approaches are presented. 1) Speaker adaptation - Step1: A multispeaker generative model is trained first with large amount of data (text/speech). Step 2: In order to clone voice of a new speaker (with few samples), the model learns the embedding of the speaker using the trained generative model learnt with large amount of data. This is what they call the speaker adaptation approach without fine tuning.  In addition to the above, one may also make the generative model tunable (along with the new speaker's embeddings). This is called 'whole model adaptation' in the paper. It is noted that the whole model adaptation produces the best speaker samples, and trains much faster than the case where only the speaker is implicitly adapted but the model remains fixed. Step 3: The adapted speaker embedding is now used for the inference task where text is passed along with speaker encoding to produce voice from the new speaker.  2) Speaker encoding - Step 1: The first step is to train the generative model along with speaker embeddings as in the speaker adaptation appraoch. The difference here is that the model also learns to produce as output an encoding for speakers. Step 2: For the voice cloning task, the model outputs a speaker encoding explicitly when presented with voice samples from the target to be cloned. Step 3: After an encoding/embedding is produced in the second step, one may use it for inference in the speaker adaptation approach where one passes in some text and the newly produced encoding to generate voice from the cloned speaker.   The paper goes on to show results to demonstrate that both approaches are viable (samples are available at https://audiodemos.github.io/ as indicated in the paper).   There are many details and minutiae in the work of very fine granularity. A few points came to the attention of this reviewer.  1) Training speaker encoding: Lines 128, 129 in Section 3.2 (Speaker encoding). Optimization challenges are mentioned while training the speaker encoding approach. The model tends to average out speaker voices to minimize overall generative loss. Instead, they device a workaround by training each speaker separately and fine tuning the model afterwards.   2) In addition to human evaluations, the architecture produces two comparison metrics to assist in the verification process.  a) Speaker classification - determining which speaker an audio sample belongs to b) Speaker verification - verifying the identity of the speaker. In other words, performing binary classification to identify whether the test audio enrolled is from the same speaker.  c) Latent space walks - While this work does not claim to explore stochastic (in the style of VAE) latent manifolds, it presents similar latent style transformations by manipulating embeddings (section 4.4 - Voice morphing via embedding manipulation). This could be useful pointers for investigations in tasks such as transforming voices.    Results: A fairly extensive set of experiments are conducted to compare quality of generated samples. Notably, the quality improves with the number of samples used for cloning (the number of samples used varies from 1-100). Also, the results consistently show that whole model adaptation is 'better' (when we listen to the samples) than single speaker embedding. The speaker embedding and encoding performance seem to be on par.   While I think that the paper's subject matter, ideas and results are truly exploring the limits of what is possible now in neural TTS, it seems that this would be an extremely difficult project to replicate owing to the complexity of the models presented and the generally pipelined nature of the space (it is not really end-to-end).   Comments on pertinent evaluation criteria:  1) Quality: Excellent. Encyclopedic presentation of current neural TTS landscape. Several novel ideas.  2) Clarity: Excellent. The paper is readable and well written.  3) Originality: Very Good. Voice cloning is an interesting (although these problems have been around for a long time) application for DNNs. The paper presents several original ideas in speaker adaptation, encoding and latent vector manipulation.  4) Significance: Excellent. The problems explored (text to speech, voice cloning, voice conversion, speaker verification etc.) in this paper are very pertinent to speech synthesis research. It demonstrates the massive progress made in recent times in DNN based TTS.