Versatile Framework for Song Generation with Prompt-based Control (2025)

Abstract: Song generation focuses on producing controllable high-quality songs based on various prompts. However, existing methods struggle to generate vocals and accompaniments with prompt-based control and proper alignment. Additionally, they fall short in supporting various tasks. To address these challenges, we introduce VersBand, a multi-task song generation framework for synthesizing high-quality, aligned songs with prompt-based control. VersBand comprises these primary models: 1) VocalBand, a decoupled model, leverages the flow-matching method for generating singing styles, pitches, and mel-spectrograms, allowing fast, high-quality vocal generation with style control. 2) AccompBand, a flow-based transformer model, incorporates the Band-MOE, selecting suitable experts for enhanced quality, alignment, and control. This model allows for generating controllable, high-quality accompaniments aligned with vocals. 3) Two generation models, LyricBand for lyrics and MelodyBand for melodies, contribute to the comprehensive multi-task song generation system, allowing for extensive control based on multiple prompts. Experimental results demonstrate that VersBand performs better over baseline models across multiple song generation tasks using objective and subjective metrics.

## introduction song generation focuses on producing complete musical pieces based on text prompts (about lyrics, melodies, singing, and music styles) and optional audio prompts. unlike singing voice synthesis (svs) @xcite , which focuses on the vocal component, or music generation @xcite for synthesizing only instrumental * equal contribution † corresponding author tracks, song generation involves synthesizing both high-quality vocals and accompaniments with highlevel prompt-based control and proper alignment in rhythm, melody, and beats @xcite . despite significant advancements in svs and music domains, generating high-quality, controllable, aligned songs remains challenging. song generation aims to enable controllable musical experiences, with broad applications ranging from entertainment videos to professional composition. as shown in figure 1 , song generation models can leverage different prompts for multiple tasks. text prompts allow for control over tasks, lyrics, melody, singing styles (like singing methods, emotion, and techniques), and music styles (like genre, tone, and instrumentation), while audio prompts enable users to input their voice or preferred music for customization. however, the few existing song generation models @xcite lack mechanisms to properly align vocals with accompaniments and fail to achieve effective control. currently, song generation encounters three major challenges: 1) limitations in high-quality vocal generation with style control. for singing style control, stylesinger @xcite conducts style transfer, while promptsinger @xcite achieves singer identity control. however, existing models have yet to generate pleasing vocals with high-level style control (like singing methods, emotion, and techniques) by text prompts, and customization with audio prompts. 2) difficulties in controllable and aligned accompaniment generation. existing music generation models @xcite ) and text-to-song models @xcite lack mechanisms for style control and aligning vocals with accompaniments in rhythm, melody, and beats. generating controllable (like genre, tone, and instrumentation) and aligned accompaniments remains challenging. 3) challenges in multi-task song generation based on various prompts. the limited existing song ## conclusions in this paper, we present versband, a multi-task song generation framework for synthesizing highquality, aligned songs with prompt-based control. we mainly design these key models for the multitask comprehensive song generation system: 1) vocalband, a decoupled model leveraging the flowmatching model for singing styles, pitches, and mel-spectrograms generation, allowing fast and high-quality vocal generation with style control.