DPP - TTS : Diversifying prosodic features of speech via determinantal point processes (2023)

Abstract: With the rapid advancement in deep generative models, recent neural Text-To-Speech(TTS) models have succeeded in synthesizing human-like speech. There have been some efforts to generate speech with various prosody beyond monotonous prosody patterns. However, previous works have several limitations. First, typical TTS models depend on the scaled sampling temperature for boosting the diversity of prosody. Speech samples generated at high sampling temperatures often lack perceptual prosodic diversity, which can adversely affect the naturalness of the speech. Second, the diversity among samples is neglected since the sampling procedure often focuses on a single speech sample rather than multiple ones. In this paper, we propose DPP-TTS: a text-to-speech model based on Determinantal Point Processes (DPPs) with a prosody diversifying module. Our TTS model is capable of generating speech samples that simultaneously consider perceptual diversity in each sample and among multiple samples. We demonstrate that DPP-TTS generates speech samples with more diversified prosody than baselines in the side-by-side comparison test considering the naturalness of speech at the same time.

## introduction in the past few years, text-to-speech (tts) models have made a lot of progress in synthesizing human-like speech @xcite @xcite @xcite . furthermore, in the latest studies, several tts models made high-quality speech samples even in the end-to-end setting without a two-stage synthesis process @xcite . based on these technical developments, tts models are now able to generate high-fidelity speech. meanwhile, human speech contains diverse prosody patterns regarding intonation, stress, and rhythm beyond the fidelity of speech. to reflect such acoustic features on generated speech, there have been many attempts to synthesize speech with rich and diverse prosodic patterns. one of the widely used approaches for prosody modeling is to exploit generative models like vaes and flow models @xcite @xcite @xcite . these generative tts models control the extent of variation in speech by sampling prior distribution with adequate temperatures. in other works, auxiliary features such as syntax information and text semantics from bert embeddings are used to enhance the prosody of speech @xcite . however, previous approaches are subject to several limitations. first, sampling with high temperatures for generating diverse prosody patterns often severely degrades the naturalness of speech. in the experiment section, we show that previous tts models often fail to generate diverse and smooth speech samples to the listeners in diverse sampling temperature settings. second, previous works for prosody modeling treat each sample independently, thereby not guaranteeing the diversity among separately generated samples. in this paper, for generating diverse speech samples, we resolve the two aforementioned limitations by adopting determinantal point processes (dpps), which have typically shown great results in modeling diversity among multiple samples in various machine learning tasks such as text summarization @xcite and recommendation systems @xcite . we devise a novel tts model equipped with the new objective function and prosody diversifying module based on dpps. through our adaptive mic objective function and dpp kernel, we can effectively generate speech samples with diverse and natural prosody. however, with the standard dpp, it is challenging to model a fine-grained level of prosody for non-monotonous speech. for more sophisticated prosody modeling, we also propose conditional dpps which utilize the conditional information for the sampling. specifically, we first segment the whole input text into more fine-grained text segments, and then sample prosodic features for the targeted segment, reflecting its neighbor segments to generate more expressive samples. in the process of adopting conditional dpps into our tts models, two technical difficulties must be handled. first, the range of targets for the sampling of conditional dpp is ambiguous. second, prosodic features usually vary in length. we resolve the first issue by extracting key segments from the input text using prosodic boundary detector (pbd). next, we resolve the second issue by adopting the similarity metric, soft dynamic time warping discrepancy (soft-dtw) @xcite , among the segments of variable lengths. experimental results demonstrate that our training methodology becomes stable with the new adaptive mic objective function, overcoming the instability of sampling-based learning. in addition, our dpp-tts is highly effective at generating more expressive speech and guaranteeing the quality of speech. we find that our model consistently gains a higher diversity score across all temperature settings than baselines. in summary, our contributions of the paper are as follows: ## dpp-tts our model dpp-tts is composed of base tts based on fastspeech2 @xcite , prosody diversifying module (pdm), and prosody boundary detector (pbd). once the base tts is trained, pdm is inserted in front of the prosody predictor and trained with the method which will be described in detail in section 3.2. we describe the main modules of dpp-tts in the following subsection 3.1. for the detailed architecture of dpp-tts, refer to appendix a. ## conclusion we propose a prosody diversifying module (pdm) that explicitly diversifies prosodic features like duration or pitch to avoid a monotonous speaking style, using conditional mic objective. previous research has indicated that well-positioned prosodic boundaries are helpful for understanding the meaning of speech @xcite . there are some possible avenues to improve our model for more realistic prosody. a more sophisticated segmentation rule that reflects human prosodic boundaries can improve the naturalness of our model. incorporating linguistic representation well-aligned with human intonation contributes to accurate prosodic boundaries. such methods will boost the performance of our model.