Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024)

Abstract: Large language models (LLMs) have successfully served as a general-purpose interface across multiple tasks and languages, while the adaptation of voice LLMs is mostly designed for specific purposes (either single-task or monolingual), where the advantages of LLMs especially for low-resource language processing and zero-shot task generalization are less exploited in the audio community. To bridge the gap, we introduce Make-A-Voice as a multi-modal voice LLM and conduct a comprehensive study on its capability to deal with multiple tasks/languages. When trained on ~200K hours of 6-language data for 4 voice generation applications, Make-A-Voice emerges notable advantages: 1) as scalable learners to improve performance with end-to-end local and global multiscale transformers; and 2) as multitask learners by adjusting prompts to share common knowledge across modalities (speech/singing) and present in-context learning abilities by generalizing to unseen tasks not explicitly train on; 3) as multilingual learners to alleviate data scarcity of low-resource languages by including rich-resource language training data. Experimental results demonstrate that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models in monolingual/cross-lingual voice generation. Audio samples are available at https://M-Voice.github.io

## introduction large language models (llms) @xcite @xcite trained on massive corpora of texts have shown their ability to perform new tasks from textual instructions or a few examples. the llm-based interfaces excel at generating text for tasks that require the modeling of complex interactions, and they are trained to predict sequences of discrete tokens, have also been adapted to continuous, audio signals @xcite @xcite . current voice llms @xcite @xcite cast voice synthesis as a language modeling task in a discrete representation space. vall-e @xcite proposes a language model approach for tts with audio codec codes as intermediate representations. @xcite introduce a hierarchical approach that combines two types of audio tokens. @xcite train a multilingual conditional codec language model to predict the acoustic token sequences of speech in different languages. despite the success achieved, most existing voice llms are designed for specific purposes (single-task or monolingual), where the advantages of llms especially for low-resource language processing and zero-shot task generalization are less exploited in the audio community. to bridge the gap, we leverage the intuition that gpts can process multiple tasks or languages as a general-purpose interface and emerge strong incontext learning capabilities. in this work, we conduct a comprehensive study on voice llms' capability to deal with multiple tasks (i.e., speech and singing voice modeling) and multiple languages (i.e., rich and low resource data). we refer to this resulting model as make-a-voice, for multimodal llms to synthesize and manipulate voice signals. make-a-voice is a decoder-only model trained on a mixture of tasks, emerging capabilities of cross-task knowledge sharing, generalization to new tasks, and alleviating data scarcity of lowresource languages. make-a-voice employs self-supervised tokens for the unified voice generation pipeline: 1) semantic tokens determine the semantic meaning given text or speech; 2) acoustic tokens provide acoustic information given various control conditions, which can be learned in a large amount of self-supervised audio-only data. make-a-voice is trained on ∼200k hours of multilingual data in 6 languages, and we introduce 4 applications: text-to-speech (tts), voice conversion (vc), singing voice synthesis (svs), and singing voice conversion (svc). experimental results demonstrate that make-a-voice achieves sota results in monolingual/cross-lingual zero-shot voice generation. subjective and objective evaluation show that make-a-voice exhibits superior audio quality and style similarity compared with baselines. the key takeaways are as follows: combined with more rich-resource language data, model training benefits from a variety of acoustic conditions on top of semantic meanings. make-a-voice excels at zero-shot transferring acoustic attributes (speaker identity, emotion, prosody) even in low-resource languages, which has witnessed a 20% sim gain and 7.9% wer drop in tts synthesis. ## voice large language models in this section, we overview the discrete voice representation, namely semantic and acoustic tokens, and then introduce the decoder-only unified voice synthesis model make-a-voice. next, we introduce the designs of the multitask learning in section 3.3 and multilingual approaches in section 3.4, as well as the scalable multi-scale transformer architecture in section 3.5. ## conclusion to bridge the gap where multiple tasks and languages are less exploited in voice llms compared to gpts, we introduced make-a-voice, a multimodal llm to synthesize and manipulate voice signals. make-a-voice took self-supervised tokens first to determine the semantic meaning and then learned acoustic information given condition signals, and we conducted a comprehensive study on its capabilities to deal with multiple tasks and languages. when trained on ∼200k hours of 6language data for 4 voice generation applications, make-a-voice emerged notable advantages as 1) as scalable learners to improve performance with end-to-end differentiable local and global multiscale transformers; and 2) as multitask learners to share common knowledge across modalities (speech/singing) and presented in-context learning abilities by performing tasks not explicitly trained on; 3) as multilingual learners to alleviate data scarcity of low-resource languages by including rich-resource language training data. experimental results demonstrated that make-a-voice achieved state-of-the-art results in monolingual/cross-lingual zero-shot voice generation. the subjective and objective evaluation showed that make-a-voice exhibited superior audio quality and similarity compared with baselines.