CM - TTS : Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models (2024)

Abstract: Neural Text-to-Speech (TTS) systems find broad applications in voice assistants, e-learning, and audiobook creation. The pursuit of modern models, like Diffusion Models (DMs), holds promise for achieving high-fidelity, real-time speech synthesis. Yet, the efficiency of multi-step sampling in Diffusion Models presents challenges. Efforts have been made to integrate GANs with DMs, speeding up inference by approximating denoising distributions, but this introduces issues with model convergence due to adversarial training. To overcome this, we introduce CM-TTS, a novel architecture grounded in consistency models (CMs). Drawing inspiration from continuous-time diffusion models, CM-TTS achieves top-quality speech synthesis in fewer steps without adversarial training or pre-trained model dependencies. We further design weighted samplers to incorporate different sampling positions into model training with dynamic probabilities, ensuring unbiased learning throughout the entire training process. We present a real-time mel-spectrogram generation consistency model, validated through comprehensive evaluations. Experimental results underscore CM-TTS’s superiority over existing single-step speech synthesis systems, representing a significant advancement in the field.

## introduction the modern neural text-to-speech (tts) system @xcite @xcite stands out for its exceptional naturalness and efficiency, proving versatile in human-computer interaction and content generation scenarios like real-time voice broadcasting and speech content creation. comprising three integral modules, the system involves a text encoder collaborating with a conditioning feature predictor, followed by an acoustic model transforming conditioning features into speech features, and a vocoder converting synthesized features into audible speech. this intricate process ensures efficient synthesis of human-like speech. from a formulation perspective, tts architecture aligns with autoregressive (ar) @xcite @xcite and non-autoregressive (nar) @xcite models. ar frameworks, using rnn models with attention mechanisms, generate spectrograms sequentially, ensuring stable synthesis but suffering from accumulated prediction errors and slower inference speeds. conversely, nar models, often based on transformer architecture @xcite , employ parallel feed-forward networks for simultaneous mel-spectrogram generation, reducing computational complexity and enabling real-time applications. various generative models, including generative adversarial networks (gans) @xcite @xcite , flow @xcite @xcite -based models, and hybrid approaches like flow with gan @xcite , contribute to high-fidelity, real-time speech synthesis. diffusion models (dms) are advanced generative models, excelling in image generation @xcite @xcite , molecular design @xcite @xcite , and speech synthesis @xcite . employing a forward diffusion process with noise addition and a parameterized reverse iterative denoising process, dms efficiently capture high-dimensional data distributions. despite their exceptional performance, the efficiency of their multi-step iterative sampling is hindered by markov chain limitations. to address these challenges, @xcite propose a tts ar-chitecture based on consistency models @xcite . this architecture achieves high audio quality through a single diffusion step, applying a consistency constraint to distill a model from a welldesigned diffusion-based teacher model. however, a drawback is the method's reliance on distillation from a teacher model, introducing complexity into the training pipeline. importantly, their proposed tts architecture is trained on the single-speaker ljspeech dataset @xcite , limiting its suitability for multi-speaker speech generation. this constraint should be considered in applications where broader speaker diversity is essential. the integration of gans into dms for tts synthesis @xcite has proven effective in minimizing the number of sampling steps during the speech synthesis process. however, this improvement comes at the cost of hindered model convergence due to the additional training required for the discriminator. some approaches enhance synthesis performance with fewer inference steps by incorporating a shallow diffusion mechanism @xcite . nonetheless, the introduction of an additional pre-trained model adds complexity to the overall architecture. we present a novel tts architecture, cm-tts, addressing current limitations without relying on a teacher model for distillation. drawing inspiration from continuous-time diffusion and consistency models, our approach frames speech synthesis as a generative consistency procedure, achieving superior quality in a single step. cm-tts eliminates the need for adversarial training @xcite or auxiliary pre-trained models @xcite . we enhance model training efficacy with weighted samplers, mitigating sampling biases. cm-tts maintains traditional diffusion-based tts benefits and introduces a few-step iterative generation, balancing synthesis efficiency and quality. experimental results confirm cm-tts outperforms other single-step speech synthesis systems in quality and efficiency, presenting a significant advancement in tts architecture. our key contributions can be summarized as follows: non-autoregressive generative models nonautoregressive generative models (nar) excel in swiftly generating output, making them ideal for real-time applications. their efficiency, derived from parallelized output generation and lack of dependence on previous results, finds applications in diverse domains like image generation and speech synthesis. gan networks have been applied in nonautoregressive speech synthesis. donahue et al. (2020) employ adversarial training and a differentiable alignment scheme for end-to-end speech synthesis. additionally, kim et al. (2021) integrate adversarial training into variational autoencoders (vae)((kingma and welling, 2019)), enhancing expressive power in speech generation. however, gans face training instability due to non-overlapping distributions between input and generated data. to address this, cm-tts incorporates diffusion model principles for improved model training and mel-spectrogram generation. diffusion models (dms) dms provide robust frameworks for learning complex high-dimensional data distributions through continuous-time diffusion processes. after surpassing gans (dhariwal and nichol, 2021) in image synthesis, dms have shown promise in speech synthesis. jeong et al. (2021) utilize a denoising diffusion framework for efficient speech synthesis, transforming noise signals into mel-spectrograms. while dms excel in data distribution modeling, they may require numerous network function evaluations (nfes) during sampling. combining diffusion modeling with traditional generative models enhances efficiency. diff-gan (liu et al., 2022b) adopts an adversarially trained model for expressive denoising distribution approximation. yang et al. (2023) use vq @xcite to transfer text features to mel-spectrograms, reducing diffusion model computational complexity. ## background: consistency models the diffusion model is distinguished by a sequential application of gaussian noise to a target dataset, followed by a subsequent reverse denoising process @xcite . this iterative methodology is designed to generate samples from an initially noisy state, effectively capturing the intrinsic structure of the data. consider the sequence of noisy data {x} @xmath0,t ] , where p 0 (x) ≡ p data (x), p t (x) approximates a gaussian distribution, and t represents the time constant. the diffusion process can be mathematically expressed as a stochastic process using following stochastic differential equation (sde). a fundamental characteristic of the sde lies in its inherent possession of a well-defined reverse process, manifested in the form of a probability flow ode @xcite . consequently, the trajectories sampled at time t follow a distribution governed by p t (x t ): the forward step induces a shift in the sample away from the data distribution, dependent on the noise level. conversely, a backward step guides the sample closer to the expected data distribution. the probability flow ode (referenced as eq. 2) for sample generation utilizes the score function ∇ log p t (x t ). obtaining the score function involves minimizing the denoising error ||f (x t , t)-x|| 2 (karras et al., 2022), where f (x t , t) is the denoiser function refining the sample x t at step t. ∇ log p t (x t ) = (f (x t , t)x t ) σ(t) 2 probability flow odes sampling follows a twostep approach: first, samples are drawn from a noise distribution, and then, a denoising process is applied using a numerical ode solver, like euler or heun @xcite . however, the sampling process from the ode solver requires a substantial number of iterations, leading to the drawback of slow inference speed. to further accelerate the sampling @xcite proposed a consistency property for the diffusion model with the following condition for any time step t and t ′ of a solution trajectory. given the aforementioned condition, one-step sampling f (x t , t ) becomes viable, as each point along the sampling trajectory of the ode is directly associated with the origin p 0 (x). for a more in-depth discussion, refer to @xcite . the consistency model is categorized into two types: consistency training or distillation from a pre-trained diffusion-based teacher model. the distillation-based approach relies on the teacher model, adding intricacy to the construction pipeline of the speech synthesis system. in this work, we opt for consistency training of the consistency model. ## cm-tts diffusion models, known for their high-quality outputs, often struggle with real-time demands in tts systems due to slow sampling. existing attempts, like diff-gan @xcite , often rely on additional adversarial training or pre-trained models for efficiency and accuracy. in this section, we discuss the architecture of cm-tts.