TCS inger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis (2025)

Abstract: Customizable multilingual zero-shot singing voice synthesis (SVS) has various potential applications in music composition and short video dubbing. However, existing SVS models overly depend on phoneme and note boundary annotations, limiting their robustness in zero-shot scenarios and producing poor transitions between phonemes and notes. Moreover, they also lack effective multi-level style control via diverse prompts. To overcome these challenges, we introduce TCSinger 2, a multi-task multilingual zero-shot SVS model with style transfer and style control based on various prompts. TCSinger 2 mainly includes three key modules: 1) Blurred Boundary Content (BBC) Encoder, predicts duration, extends content embedding, and applies masking to the boundaries to enable smooth transitions. 2) Custom Audio Encoder, uses contrastive learning to extract aligned representations from singing, speech, and textual prompts. 3) Flow-based Custom Transformer, leverages Cus-MOE, with F0 supervision, enhancing both the synthesis quality and style modeling of the generated singing voice. Experimental results show that TCSinger 2 outperforms baseline models in both subjective and objective metrics across multiple related tasks.

## introduction zero-shot singing voice synthesis (svs) aims to generate high-quality singing voices with unseen multi-level styles based on audio or textual prompts @xcite . this field has found widespread potential applications in professional music composition and short video dubbing. zero-shot svs involves using an acoustic model to leverage lyrics and musical notations for content modeling, while audio or textual prompts can control singing styles. finally, a vocoder is employed to synthesize the target singing voice. although traditional svs tasks @xcite @xcite have made significant strides, there is an increasing demand for more customizable experiences. this includes not only zero-shot style transfer by audio prompts @xcite , but also the need to leverage natural language textual prompts for multilevel style control. textual prompts can influence global timbre by specifying the singer's gender and vocal range. additionally, they can control broader aspects of singing style, such as vocal techniques (e.g., bel canto) and emotional expression (e.g., happy or sad), as well as segment-or word-level techniques (e.g., mixed voice or falsetto). audio prompts, in addition, enable the target to learn these consistent multi-level styles while incorporating accent, pronunciation, and transitions. however, current models still struggle to effectively implement style transfer and style control based on various prompts in zero-shot scenarios. consequently, achieving a natural, stable, and highly controllable generation remains a significant challenge. currently, customizable multilingual zero-shot svs faces two major challenges: 1) existing svs models heavily rely on phoneme and note boundary annotations, which limits their robustness. datasets like opencpop @xcite depend on mfa and human-ear alignment, which introduces significant errors at the boundaries. additionally, these svs models often produce poor transitions between phonemes and notes, especially in zeroshot scenarios, where this issue becomes even more pronounced. @xcite introduces a melody-unsupervised model to reduce reliance on boundary annotations. however, the unsupervised approach results in lower synthesis quality and cannot ensure smooth transitions at the boundaries. 2) existing svs models with style transfer and style control lack effective multi-level style con-trol through diverse prompts. tcsinger @xcite achieves style control using specified labels or audio prompts. however, it still cannot cover a wider range of applications with more flexible prompts, including natural language textual, speech, or singing prompts. moreover, its capability in style control is still quite limited. to address these challenges, we introduce tc-singer 2, a multi-task multilingual zero-shot svs model with style transfer and style control based on various prompts. tcsinger 2 enables effective style control using natural language textual, speech, or singing prompts. to achieve smooth and robust phoneme/note boundary modeling, we design the blurred boundary content (bbc) encoder. this encoder predicts duration, extends content embedding, and applies masking to phoneme and note boundaries to facilitate smooth transitions and ensure robustness. furthermore, to extract aligned representations from singing, speech, and textual prompts, we propose the custom audio encoder based on contrastive learning, extending the model's applicability to a broader range of related tasks. in addition, to generate high-quality and highly controllable singing voices, we introduce the flow-based custom transformer. within this framework, we utilize cus-moe, which, depending on the language and textual or audio prompt, selects different experts to achieve better synthesis quality and style modeling. moreover, we incorporate additional supervision using f0 information to enhance the expressiveness of the synthesized output. our experimental results show that tcsinger 2 outperforms other baseline models in synthesis quality, singer similarity, and style controllability across various tasks, including zero-shot style transfer, cross-lingual style transfer, multi-level style control, and speech-to-singing (sts) style transfer. our main contributions can be summarized as: to address this, we employ blurred boundaries. ## method in this section, we first provide an overview of the proposed tcsinger 2. next, we describe several key components, including the blurred boundary content (bbc) encoder, custom audio encoder, and flow-based custom transformer. finally, we detail the training and inference processes. ## conclusion in this paper, we present tcsinger 2, a multilingual, multi-task, zero-shot singing voice synthesis model with advanced style transfer and style control capabilities based on various prompts. to ensure smooth and robust phoneme/note transitions, we introduce the blurred boundary content encoder, which applies flexible boundary masking on phoneme and note boundaries for seamless transitions. for aligned representations across singing, speech, and textual prompts, we propose the custom audio encoder using contrastive learning, broadening the model's applicability to a wide range of tasks. moreover, we also introduce the flow-based custom transformer to stably and fastly generate high-quality, highly controllable singing voices. the model employs cus-moe and f0 supervision to optimize synthesis quality and style modeling. our experimental results show that tcsinger 2 outperforms other baseline models in synthesis quality, singer similarity, and style controllability across various related tasks, including zero-shot style transfer, cross-lingual style transfer, multi-level style control, and sts style transfer.