On the Semantic Latent Space of Diffusion-Based Text-To-Speech Models (2024)

Abstract: The incorporation of Denoising Diffusion Models (DDMs) in the Text-to-Speech (TTS) domain is rising, providing great value in synthesizing high quality speech. Although they exhibit impressive audio quality, the extent of their semantic capabilities is unknown, and controlling their synthesized speech’s vocal properties remains a challenge. Inspired by recent advances in image synthesis, we explore the latent space of frozen TTS models, which is composed of the latent bottleneck activations of the DDM’s denoiser. We identify that this space contains rich semantic information, and outline several novel methods for finding semantic directions within it, both supervised and unsupervised. We then demonstrate how these enable off-the-shelf audio editing, without any further training, architectural changes or data requirements. We present evidence of the semantic and acoustic qualities of the edited audio, and provide supplemental samples: https://latent-analysis-grad-tts.github.io/speech-samples/.

## introduction denoising diffusion models (ddms) @xcite have emerged as a powerful generative tool across a broad variety of tasks and domains. in particular, text-to-speech (tts) systems based on diffusion have shown high-quality speech generation capabilities @xcite . although these exhibit improved quality, the extent to which they capture semantic information is yet to be uncovered, and the ability to control the vocal properties (e.g. volume, pitch, gender) of their generated speech is limited. uncovering the semantic capabilities of tts diffusion models will allow editing the properties of synthesized speech, which is essential in real-world applications, such as human-machine interaction. ## conclusions in this paper, we identify the semantic properties of the latent space of diffusion-based tts models, referred to as h-space. we develop supervised and unsupervised methods for finding interpretable directions in that space, and provide empirical qualitative evidence for their semantic quality. moreover, the proposed latent space editing methods preserve and even enhance the acoustic quality of the generated samples. this study presents evidence regarding specific vocal attribute manipulation, such as gender or intensity. however, the presented method can be applied to any vocal attribute present in the data.