M obile S peech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speech (2024)

Abstract: Zero-shot text-to-speech (TTS) has gained significant attention due to its powerful voice cloning capabilities, requiring only a few seconds of unseen speaker voice prompts. However, all previous work has been developed for cloud-based systems. Taking autoregressive models as an example, although these approaches achieve high-fidelity voice cloning, they fall short in terms of inference speed, model size, and robustness. Therefore, we propose MobileSpeech, which is a fast, lightweight, and robust zero-shot text-to-speech system based on mobile devices for the first time. Specifically: 1) leveraging discrete codec, we design a parallel speech mask decoder module called SMD, which incorporates hierarchical information from the speech codec and weight mechanisms across different codec layers during the generation process. Moreover, to bridge the gap between text and speech, we introduce a high-level probabilistic mask that simulates the progression of information flow from less to more during speech generation. 2) For speaker prompts, we extract fine-grained prompt duration from the prompt speech and incorporate text, prompt speech by cross attention in SMD. We demonstrate the effectiveness of MobileSpeech on multilingual datasets at different levels, achieving state-of-the-art results in terms of generating speed and speech quality. MobileSpeech achieves RTF of 0.09 on a single A100 GPU and we have successfully deployed MobileSpeech on mobile devices. Audio samples are available at https://mobilespeech.github.io/

## introduction in recent years, remarkable progress has been made in the development of text-to-speech (tts) technology @xcite @xcite . with advancements in this field, current state-of-the-art tts systems have demonstrated the ability to generate high-quality voices of unseen speakers while maintaining consistency in tone and rhythm @xcite . however, these systems are often trained on datasets of only a few hundred hours and often require fine-tuning with dozens of sentences during the inference stage, which significantly limits the generative capabilities of the models and is time-consuming and not user-friendly. with the recent development of large-scale language models @xcite and the increase in training data for speech synthesis from hundreds of hours @xcite to tens of thousands of hours @xcite , current tts models exhibit powerful zero-shot capabilities, enabling the cloning of unseen speakers' voices in just a few seconds @xcite @xcite @xcite @xcite @xcite . however, these models primarily focus on incontext abilities and fail to address issues related to inference speed and model deployment parameters. additionally, all existing zero-shot tts models are cloud-based and lack a mobile-based deployment approach. mobilespeech is the first zero-shot tts synthesis system that can be deployed on mobile devices. firstly we think deployable mobile zero-shot tts models aim to achieve the following goals: to achieve these objectives, we have made our best efforts to reproduce several state-of-the-art generative models, which will be elaborated in the subsequent section on related work. we found current work is not applicable to mobile devices. ultimately, based on the natspeech framework @xcite and the mask generation module @xcite , we additionally design the smd module and the speaker prompt component, taking into account the discrete codec architecture of speech. in summary, mobile-speech contributes as follows: ## mobilespeech in this section, we will first introduce the overall structure of mobilespeech, followed by a detailed focus on the speech mask codec decoder module and the speaker prompt module. furthermore, we will proceed to elaborate on the specific intricacies of mobilespeech's training and inference processes, with the explicit details of the model being presented in appendix c. ## conclusion in this paper, we propose mobilespeech, a fast and lightweight framework for mobile zero-shot textto-speech. mobilespeech achieves a unique combination of audio naturalness and similarity performance, comparable to other zero-shot tts systems on english corpora, while significantly enhancing inference speed through its distinctive smd and speaker modules. mobilespeech stands as the first real-time zero-shot tts system deployable at the edge, demonstrating state-of-the-art performance on chinese corpora through extensive training on large-scale chinese language data. we not only validate the significance of each module in mo-bilespeech through ablation experiments but also analyze the effects of different configurations and iterations to adapt to various deployment scenarios in mobile environments.