S tyle TTS - ZS : Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusion (2025)

Abstract: The rapid development of large-scale text-to-speech (TTS) models has led to significant advancements in modeling diverse speaker prosody and voices. However, these models often face issues such as slow inference speeds, reliance on complex pre-trained neural codec representations, and difficulties in achieving naturalness and high similarity to reference speakers. To address these challenges, this work introduces StyleTTS-ZS, an efficient zero-shot TTS model that leverages distilled time-varying style diffusion to capture diverse speaker identities and prosodies. We propose a novel approach that represents human speech using input text and fixed-length time-varying discrete style codes to capture diverse prosodic variations, trained adversarially with multi-modal discriminators. A diffusion model is then built to sample this time-varying style code for efficient latent diffusion. Using classifier-free guidance, StyleTTS-ZS achieves high similarity to the reference speaker in the style diffusion process. Furthermore, to expedite sampling, the style diffusion model is distilled with perceptual loss using only 10k samples, maintaining speech quality and similarity while reducing inference speed by 90%. Our model surpasses previous state-of-the-art large-scale zero-shot TTS models in both naturalness and similarity, offering a 10-20× faster sampling speed, making it an attractive alternative for efficient large-scale zero-shot TTS systems. The audio demo, code and models are available at https://styletts-zs.github.io/.

## introduction recent advancements in text-to-speech (tts) technology have achieved remarkable progress, bringing tts systems close to, or even surpassing, human-level performance on various benchmark datasets @xcite @xcite . with studio-level tts capabilities * work done in part during an internship at descript. nearly perfected, there is a growing demand for more sophisticated tasks such as diverse and personalizable zero-shot speaker adaptation @xcite . these tasks present a significant challenge due to the need to replicate the unique characteristics and prosodic variations of a vast array of speakers without extensive training data for each individual. although there have been rapid developments in zero-shot adaptation, driven by large-scale modeling techniques in large language models (llms) @xcite @xcite @xcite , high-quality discrete audio codecs @xcite @xcite , and diffusion-based models @xcite @xcite @xcite , current models face crucial limitations. many large-scale speech synthesis models rely on auto-regressive modeling @xcite @xcite @xcite @xcite , which results in slower scaling of inference speed as the length of the target speech increases. alternatively, diffusion-based models are used for building largescale speech synthesis models @xcite @xcite @xcite . however, since these models require iterative refinement to produce high-quality results, they also suffer from efficiency issues. moreover, these models often depend on pre-trained neural codecs not specifically designed for tts tasks @xcite , limiting their ability to naturally model diverse human speech, which encompasses a wide range of speaking styles that can be challenging to control with existing codecs. in this work, we introduce styletts-zs, an innovative approach to diverse speech synthesis that aims to address these limitations. our model decomposes human speech into a global style vector derived from a speaker prompt and prompt-aligned text embeddings that encapsulate the timbre and acoustic features of the speech. additionally, it includes a fixed-length time-varying style vector that encodes the diverse prosodic variations, such as pitch and duration changes over time. by carefully designing the bottleneck for the style vector space with vector quantization @xcite and multimodal adversarial training @xcite , we can reconstruct speech with high fidelity. we then train a diffusion model @xcite to sample the time-varying style vector, effectively modeling the diversity of prosodic variations conditioned on the speaker prompt. this results in efficient latent diffusion, as the latent variable is a fixed-length style vector. the simplicity and efficiency of our latent space enable us to distill the teacher diffusion model into a student model using only 10k samples. this distillation maintains diversity and similarity to the prompt speaker while reducing inference to one step. our evaluation results demonstrate the effectiveness of styletts-zs. when trained on the small-scale libritts dataset @xcite , our model surpasses several public zero-shot tts baseline models. furthermore, when trained on the large-scale librilight dataset @xcite , comprising 60k hours of data, our model performs comparably to previous largescale state-of-the-art (sota) tts models in similarity and even surpasses them in naturalness for unseen speakers on the librispeech dataset using only a 3-second reference speaker prompt. remarkably, we achieve this with nearly 10-20 times faster inference speeds compared to previous sota models, showcasing its real-time applicability. ## styletts-zs styletts-zs consists of four modules: acoustic synthesizer, prosody autoencoder, time-varying style diffusion, and multimodal discriminators. we detail these four modules in the following sections with an overview of our framework in figure 1 and implementation details in appendix d and c. ## limitations although our model outperforms previous state-ofthe-art models, it still has not achieved human-level performance for zero-shot tts with unseen speakers, as models trained on small-scale datasets for seen speakers have achieved @xcite . moreover, our model prioritizes speed over quality, meaning the acoustic synthesizer does not benefit from the latest generative modeling developments with iterative refinements that could further enhance speaker fidelity to the prompt speaker. since recent works have demonstrated that diffusion-based models with iterative sampling can achieve close-to-human similarity to the prompt speaker @xcite @xcite , it is worth investigating the potential of replacing our ganbased acoustic synthesizer with diffusion-based one while preserving the superior inference speed through further distillation. additionally, the models were trained on english audiobook reading datasets rather than audio from more diverse, realworld environments and other languages. future research should explore higher-quality generation methods to improve quality without compromising speed, expand training to include more diverse datasets and languages to enhance generalizability, and further refine prosody modeling to achieve even more natural and expressive speech synthesis. still, our findings indicate that styletts-zs is a robust and efficient solution for zero-shot tts synthesis, with promising potential for applications on large-scale real-world data and future research directions. ## ethical concerns the capabilities of our model can bring potential negative impacts. the ability to generate highquality, recognizable speech could be misused for voice spoofing, posing risks to personal and financial security. there is also the potential for creating convincing deepfake audio, which can be used maliciously in misinformation campaigns, fraud, or defamation. to mitigate these risks, we recommend controlled access to the model, with strict licensing agreements that require users to obtain consent from individuals whose voices are being cloned. establishing ethical guidelines and usage policies is essential to prevent misuse. additionally, encouraging collaboration in the research community to develop and improve deepfake detection technologies and incorporating mechanisms to watermark or trace synthetic audio to ensure accountability and traceability are necessary steps. time-varying style conditioned on the input text t and prompt x ′ . we use the ℓ 1 distance of the decoded duration (l dur ), pitch (l f 0 ) and energy (l n ) between teacher and student models from the same input noise conditioned on unseen text and speaker prompts as the metric to measure the distillation performance. we tested the model performance using the guidance ω = 5 as during our inference process. for the consistency distillation (cd) baseline, we follow algorithm 1 in @xcite for guided distillation. we use the ddim solver with noise schedule specified in section 3.3, ℓ 1 norm for distance, ema rate µ = 0.999943 as in @xcite and guidance scale range ω @xmath0, ω @xmath1. for the adversarial diffusion distillation (add) baseline, we use the architecture of multimodal prosody discriminator as the discriminator that takes the denoised time-varying style as input and text embeddings t and prompt x ′ as conditions. we set τ @xmath2. we initialized the student network with the teacher network for both cd and add baselines as in @xcite . for the case of the simulationbased approach without pre-trained initialization, we also initialized our student network with the teacher network for fair comparison. we trained our model for 100k steps with a batch size of 32. as shown in