U ni S peaker: A Unified Approach for Multimodality-driven Speaker Generation (2025)

Abstract: While recent advances in reference-based speaker cloning have significantly improved the authenticity of synthetic speech, speaker generation driven by multimodal cues such as visual appearance, textual descriptions, and other biometric signals remains in its early stages. To pioneer truly multimodal-controllable speaker generation, we propose UniSpeaker, the first framework supporting unified voice synthesis from arbitrary modality combinations. Specifically, self-distillation is firstly applied to a large-scale speech generation model for speaker disentanglement. To overcome data sparsity and one-to-many mapping challenges, a novel KV-Former based unified voice aggregator is introduced, where multiple modalities are projected into a shared latent space through soft contrastive learning to ensure accurate alignment with user-specified vocal characteristics. Additionally, to advance the field, the first Multimodal Voice Control (MVC) benchmark is established to evaluate voice suitability, diversity, and quality. When tested across five MVC tasks, UniSpeaker is shown to surpass existing modality-specific models. Speech samples and the MVC benchmark are available at https://UniSpeaker.github.io .

## introduction in recent years, the field of speech generation has seen remarkable progress @xcite , enabling the generated speech to closely resemble the actual recordings. however, traditional zero-shot speech synthesis still faces limitations, particularly in scenarios like virtual character voiceovers, where ideal reference speech may be unavailable @xcite . therefore, the focus of voice control in generative models need to shift from speaker cloning to speaker generation. unlike cloning voices from reference sam-* corresponding author task model voice description modality speech text face attribute one-to-many mapping due to cross-modal information gaps, it is challenging to fully reconstruct true voice characteristics from other modalities alone. consequently, a single face or text description may correspond to multiple plausible timbres during inference, requiring further voice attributes editing @xcite to meet the desired requirements. however, current speaker generation models struggle with one-to-many ambiguity in absolute voice descriptions and lack effective coordination among diverse modalities like speech, text, face, and attributes. to overcome these challenges, we propose unispeaker, which can first utilize absolute voice descriptions to generate a coarse speaker timbre, and then enables iterative refinement through relative attribute editing to better align with user expectations. furthermore, unispeaker introduces a coordinated fusion of multimodal inputs, facilitating consistent and controllable speaker generation. specifically, we propose a unified multimodal voice aggregator (mva) that aligns multimodal inputs into a unified voice space. the core of mva is the kv-former, which selectively integrates relevant details from each modality, capturing mutual information between modalities to form a coherent representation. the output is then fed to a generative model for voice control and speaker embedding alignment. to handle the correlation between voice characteristics of different speakers, soft contrastive learning (softcl) is proposed during training, which loosens strict one-to-one constraints and leverages intra-modal discriminative information. inspired by imagebind @xcite , our speech-anchoring mechanism enables cross-modal alignment without requiring parallel data, addressing data scarcity and ensuring diverse voice characteristics. in addition, while large-scale speech generation models excel in voice control, scalable multimodal integration remains underexplored. to address this, we apply self-distillation techniques @xcite on cosyvoice @xcite to improve voice disentanglement and ensure the versatility across diverse multimodal inputs. due to the lack of publicly accessible benchmarks, a multimodal voice control (mvc) benchmark was developed, evaluating speech on voice suitability, voice diversity, and speech quality across 5 tasks for diverse multimodal voice characterics. using this benchmark, unispeaker was assessed and showed superior performance over previous modality-specific models across all tasks. ## methods in this section, we first review the basic speech generation backbone, then introduce how multimodal voice descriptions are integrated into a pre-trained large speech generation model. ## conclusion in this paper, we propose unispeaker, a speech generation model that leverages multimodal voice description for voice control. with a unified voice aggregator and designed training strategies, unispeaker outperforms previous modality-specific models across five tasks, generating voices that better match the input descriptions. in the future, we will explore how to more effectively utilize multiple voice descriptions of different modalities for one speaker simultaneously and apply our method on other more modalities for voice control.