k NN Retrieval for Simple and Effective Zero-Shot Multi-speaker Text-to-Speech (2025)

Abstract: While recent zero-shot multi-speaker text-to-speech (TTS) models achieve impressive results, they typically rely on extensive transcribed speech datasets from numerous speakers and intricate training pipelines. Meanwhile, self-supervised learning (SSL) speech features have emerged as effective intermediate representations for TTS. Further, SSL features from different speakers that are linearly close share phonetic information while maintaining individual speaker identity. In this study, we introduce kNN-TTS, a simple and effective framework for zero-shot multi-speaker TTS using retrieval methods which leverage the linear relationships between SSL features. Objective and subjective evaluations show that our models, trained on transcribed speech from a single speaker only, achieve performance comparable to state-of-the-art models that are trained on significantly larger training datasets. The low training data requirements mean that kNN-TTS is well suited for the development of multi-speaker TTS systems for low-resource domains and languages. We also introduce an interpolation parameter which enables fine-grained voice morphing. Demo samples are available at https://idiap.github.io/knn-tts .

## introduction neural text-to-speech (tts) synthesis has advanced significantly in recent years, achieving a level of naturalness comparable to human speech. and allowing for an increasingly expressive range of outputs @xcite . neural tts systems can be categorized into two-stage and single-stage pipelines. two-stage models convert text or phonemic features into acoustic features and then use a vocoder to generate waveforms. these models can suffer from error propagation and limitations due to their dependence on low-level features like mel-spectrograms @xcite . single-stage models aim to address these issues by streamlining this process into an end-toend framework @xcite , but they may face oversmoothing, mispronunciations, and reduced flexibility due to the lack of explicit linguistic information and entangled latent representations @xcite . recent research combines the strengths of both approaches by using self-supervised learning (ssl) speech representations as intermediate elements in two-stage models @xcite @xcite . these representations help improve word error rates, pronunciation of out-of-vocabulary words @xcite , and robustness to noise @xcite . in practice, end-user applications may need to synthesize speech in the voices of multiple speakers. collecting high quality speech data and building a tts model for each target voice is a challenging problem. as a result, there has been a growing interest in zero-shot multi-speaker tts systems which can synthesize speech in an unseen speaker's voice based on short reference samples. state-of-the-art models such as xtts @xcite and hierspeech++ @xcite demonstrate impressive quality and similarity to unseen speakers. to produce varied voices, these models condition the output on style embeddings, which are extracted from a reference audio sample via a speaker encoder. however, these models require end-to-end training on thousands of hours of transcribed audio data from a large number of speakers to generalize effectively. simultaneously, knn-vc @xcite has emerged as a promising any-to-any voice conversion method, leveraging ssl features for zero-shot conversion. it uses a knn algorithm to match frames from the source speaker with the target speaker's representations, adjusting the speaker identity while preserving speech content. this approach is similar to retrieval-augmented generation (rag) techniques used in deep generative models such as language models @xcite and image generators @xcite . these methods have been effectively used in these fields to enhance accuracy and reliability, as well as to enable style transfer by steering model outputs to mirror characteristics of a retrieval database @xcite . in this work, we investigate whether retrievalbased methods can be similarly applied to tts for style-transfer, to achieve effective zero-shot multi-speaker capabilities. additionally, we explore whether these methods can reduce data requirements for the development of a robust zeroshot multi-speaker tts system. this paper's key contributions can be summarized as follows: code, models, and demo samples are publicly available at https://idiap.github.io/knn-tts . ## results and analysis results are presented in figure 2 illustrates the results of the voice morphing experiment. we can observe that the similarity of the outputs to the target speaker gradually increases as λ rises, demonstrating the ability to finely blend source and target styles and suggests the potential to combine multiple target styles. ## discussion and conclusions state-of-the-art zero-shot multi-speaker tts models rely on large datasets of transcribed speech from thousands of speakers for training. in this paper, we demonstrated that by leveraging retrieval methods and ssl features, we can develop a simple and lightweight tts system that achieves a comparable level of naturalness and similarity to leading approaches while being trained on transcribed data from only a single speaker. we further showed that fine-grained voice morphing can be achieved using an interpolation parameter. this indicates that this technique, which is originally inspired from other domains such as language modeling @xcite and machine translation @xcite , can be applied in the context of tts. the simplicity of the training process is a main advantage of our approach: only the text-to-ssl model needs training, and it can be trained on transcribed data from one speaker. in conjunction with the knn approach's cross-lingual capability @xcite , this is particularly appealing for extending the model to new languages with less resources, a direction open for future work. we also showed that the framework can be implemented using different text-to-ssl architectures, allowing for model swapping to leverage different benefits. our implementations notably demonstrated efficiency in terms of parameters, memory usage, and runtime speed in the case of glowknn-tts, even without optimizing the retrieval process.