Prosody- TTS : Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech (2023)

Abstract: Expressive text-to-speech aims to generate high-quality samples with rich and diverse prosody, which is hampered by dual challenges : 1) prosodic attributes in highly dynamic voices are difficult to capture and model without intonation; and 2) highly multimodal prosodic representations cannot be well learned by simple regression (e.g., MSE) objectives, which causes blurry and over-smoothing predictions. This paper proposes Prosody-TTS, a two-stage pipeline that enhances prosody modeling and sampling by introducing several components: 1) a self-supervised masked autoencoder to model the prosodic representation without relying on text transcriptions or local prosody attributes, which ensures to cover diverse speaking voices with superior generalization; and 2) a diffusion model to sample diverse prosodic patterns within the latent space, which prevents TTS models from generating samples with dull prosodic performance. Experimental results show that Prosody-TTS achieves new state-of-the-art in text-to-speech with natural and expressive synthesis. Both subjective and objective evaluation demonstrate that it exhibits superior audio quality and prosody naturalness with rich and diverse prosodic attributes. Audio samples are available at https://improved_prosody.github.io

## introduction text-to-speech (tts) @xcite @xcite aims to generate human-like audios using text and auxiliary conditions, which attracts broad interest in the machine learning community. tts models have been extended to more complex scenarios, requiring more natural and expressive voice generation with improved prosody modeling @xcite @xcite . a growing number of applications, such as personalized voice as-sistants and game commentary, have been actively developed and deployed to real-world applications. expressive text-to-speech aims to generate samples with natural, rich, and diverse prosodic attributes (e.g., duration, pitch, and energy), which is challenged by two major obstacles: 1) prosody patterns @xcite in human speech are often very sparse, which are difficult to capture and model without supervision signals (i.e., detailed transcriptions); 2) machine learning models @xcite usually learn a mean distribution over input data, resulting a dull prediction with prosody learners which fails to produce natural and diverse prosodic styles in the generated speech. although recent studies @xcite @xcite have proposed several ways to enhance prosody for high-fidelity tts, there still exist dual challenges: to address the above dual challenges for prosody-enhanced expressive text-to-speech, we propose prosody-tts, a two-stage tts pipeline that improves both prosody modeling and sampling by introducing several novel designs: a diffusion model is explored to bridge tts inputs (i.e., text and target speaker) and speaking prosody in latent space. we formulate the generative process with multiple conditional diffusion steps, and thus we expect our model to exhibit better diversity and prevent generating samples with dull prosodic performance. ## prosody-tts in this section, we first overview the prosody-tts framework, introducing several critical designs with prosody masked autoencoder (prosody-mae), latent diffusion model, and the vector quantization layer. finally, we present the pre-training, training, and inference pipeline, which supports high-fidelity speech synthesis with natural, rich, and diverse prosodic attributes. ## conclusion in this work, we propose prosody-tts, improving prosody with masked autoencoder and conditional diffusion model for expressive text-to-speech. to tackle dual challenges of prosody modeling and sampling, we design a two-stage pipeline to enhance high-quality synthesis with prosperous and diverse prosody: 1) prosody-mae was introduced to pre-train on large-scale unpaired speech datasets to capture prosodic representations without relying on text transcriptions. it ensured that the model covered diverse speaking voices and avoided inevitable error. 2) the latent diffusion model was adopted to produce diverse patterns within the learned prosody space. it broke the generation process into several conditional diffusion steps, avoiding generating samples with dull prosodic performance. experimental results demonstrated that prosody-tts promoted prosody modeling and synthesized highfidelity speech samples, achieving new state-ofthe-art results with outperformed audio quality and prosody expressiveness. for future work, we will further extend prosody-tts to more challenging scenarios such as multilingual prosody learning. we envisage that our work serve as a basis for future prosody-aware tts studies.