M ulti V erse: Efficient and Expressive Zero-Shot Multi-Task Text-to-Speech (2024)

Abstract: Text-to-speech (TTS) systems that scale up the amount of training data have achieved significant improvements in zero-shot speech synthesis. However, these systems have certain limitations: they require a large amount of training data, which increases costs, and often overlook prosody similarity. To address these issues, we propose MultiVerse, a zero-shot multi-task TTS system that is able to perform TTS or speech style transfer in zero-shot and cross-lingual conditions. MultiVerse requires much less training data than traditional data-driven approaches. To ensure zero-shot performance even with limited data, we leverage source-filter theory-based disentanglement, utilizing the prompt for modeling filter-related and source-related representations. Additionally, to further enhance prosody similarity, we adopt a prosody modeling approach combining prompt-based autoregressive and non-autoregressive methods. Evaluations demonstrate the remarkable zero-shot multi-task TTS performance of MultiVerse and show that MultiVerse not only achieves zero-shot TTS performance comparable to data-driven TTS systems with much less data, but also significantly outperforms other zero-shot TTS systems trained with the same small amount of data. In particular, our novel prosody modeling technique significantly contributes to MultiVerse’s ability to generate speech with high prosody similarity to the given prompts.

## introduction deep learning-based text-to-speech (tts) has advanced to the point of synthesizing human-like speech @xcite . however, recent research has been extended beyond this scope, exploring various ways of broadening the application of speech synthesis models. representative tasks include synthesizing unseen speaker's speech, known as zero-shot tts @xcite , generating speech in a language that the monolingual target speaker has not seen, referred to as cross-lingual tts @xcite @xcite , and transferring the prosody of a speech reference to a target speaker, known as speech style transfer @xcite . furthermore, recent tts research has integrated the zeroshot task with cross-lingual or style transfer tasks @xcite . to expand tts applications in zero-shot conditions, it is crucial to ensure generalization across various speech components, such as content, style, and speaker identity. disentangled representations facilitate this by enabling the system to capture interpretable and controllable features, thereby improving generalization of tts systems through the learning of these individual components @xcite . however, due to the often entangled nature of these components, separately learning their general characteristics remains challenging. data-driven methods are a representative approach to learning generalized acoustic components from large-scale speech datasets. specifically, these methods scale up the tts system to train on massive datasets, making them robust under unseen conditions @xcite @xcite . additionally, @xcite ; @xcite ; @xcite adopt disentangled modeling to separately learn acoustic components. disentangled modeling contributes to ensure generalization in each component by independently encapsulating interpretable elements. however, a significant amount of training dataset is required as the decoder needs to learn the relationships between disassembled elements. preparing large-scale data is especially challenging for minority languages. moreover, even with largescale training data, there is still potential room for improvement in prosody similarity in zero-shot scenarios. in this paper, we introduce a multi-task tts system, called multiverse, enabling speech synthesis and speech style transfer in zero-shot and crosslingual conditions, requiring significantly less data compared to the data-driven approaches and featuring enhanced prosody modeling. multiverse enhances training efficiency by source-filter theorybased decomposed modeling @xcite . specifically, multiverse decomposes speech generation into filter-and source-related representation generation, with prompt speech utilized in the modeling of each representation. notably, both representations result in features that have a similar distribution to the mel-spectrogram @xcite . therefore, it is suitable for the decoder to learn the interdependent relationship between the representations even with small data. furthermore, multiverse introduces an effective style modeling method. while several zero-shot models have modeled either the acoustic features @xcite or prosody latents to capture speech style @xcite , multiverse utilize both autoregressive (ar) based acoustic feature modeling and non-ar based prosody modeling. we evaluate multiverse's performance in zeroshot tts tasks under various conditions. regarding language, we evaluate both intra-and crosslingual speech synthesis and measure naturalness and similarity in neutral and expressive speech style. additionally, we compare our model with a large-scale tts model using data-driven methods and a speech style transfer model. evaluation results demonstrate that multiverse has the following advantages: (1) zero-shot intra-and cross-lingual tts is achievable with a small amount of training data. multiverse can achieve similar zero-shot synthesis in both timbre and prosody with only 1 60 of the training data compared to vall-e @xcite . ## multiverse multiverse models speech by disassembling it into two components: filter and source. the proposed model comprises three main modules: (1) an acoustic model based on the source-filter theory @xcite that generates mel-spectrograms given text and speech prompts, (2) an ar prosody predictor that predicts prosody-related acoustic features (duration, pitch, energy) from input conditions, and ## conclusion this paper introduces multiverse, an efficient and expressive zero-shot multi-task tts system designed to address the limitations of existing zeroshot tts systems that depend on large-scale training datasets. multiverse employs a structure that disentangles speech components into filter and source representations: this structure contributes to achieving zero-shot tts performance comparable to data-driven tts approaches, even with a small amount of data. additionally, it enhances prosody similarity through a hybrid prosody modeling that combines both autoregressive and nonautoregressive mechanisms. quantitative and qualitative evaluations across various language and style demonstrate that multiverse excels in zero-shot, cross-lingual tts, and speech style transfer.