TCS inger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control (2024)

Abstract: Zero-shot singing voice synthesis (SVS) with style transfer and style control aims to generate high-quality singing voices with unseen timbres and styles (including singing method, emotion, rhythm, technique, and pronunciation) from audio and text prompts. However, the multifaceted nature of singing styles poses a significant challenge for effective modeling, transfer, and control. Furthermore, current SVS models often fail to generate singing voices rich in stylistic nuances for unseen singers. To address these challenges, we introduce TCSinger, the first zero-shot SVS model for style transfer across cross-lingual speech and singing styles, along with multi-level style control. Specifically, TCSinger proposes three primary modules: 1) the clustering style encoder employs a clustering vector quantization model to stably condense style information into a compact latent space; 2) the Style and Duration Language Model (S&D-LM) concurrently predicts style information and phoneme duration, which benefits both; 3) the style adaptive decoder uses a novel mel-style adaptive normalization method to generate singing voices with enhanced details. Experimental results show that TCSinger outperforms all baseline models in synthesis quality, singer similarity, and style controllability across various tasks, including zero-shot style transfer, multi-level style control, cross-lingual style transfer, and speech-to-singing style transfer.

## introduction singing voice synthesis (svs) aims to generate high-quality singing voices using lyrics and musical notations, attracting broad interest from both industry and academic communities. the pipeline of traditional svs systems involves an acoustic model to transform musical notations and lyrics into mel-* corresponding author spectrograms, which are subsequently synthesized into the target singing voice using a vocoder. recent years have seen significant advancements in svs technology @xcite @xcite @xcite . however, the growing demand for personalized and controllable singing experiences presents challenges for current svs models. unlike traditional svs tasks, zero-shot svs with style transfer and style control seeks to generate high-quality singing voices with unseen timbres and styles from audio and text prompts. this approach can be extended to more personalized and controllable applications, such as dubbing for entertainment short videos or professional music composition. personal singing styles mainly include singing method (like bel canto), emotion (happy and sad), rhythm (including the stylistic handling of individual notes and transitions between them), techniques (such as falsetto), and pronunciation (like articulation). despite this, traditional svs methods lack the necessary mechanisms to effectively model, transfer, and control these personal styles. their performance tends to decline for unseen singers, as these methods generally assume that target singers are identifiable during the training phase @xcite . presently, zero-shot svs with style transfer and style control primarily faces two major challenges: 1) the multifaceted nature of singing styles presents a substantial challenge for comprehensive modeling, as well as effective transfer and control. previous approaches use pre-trained models to capture styles @xcite . stylesinger @xcite uses a residual quantization (rq) model to capture styles. however, these models focus on limited aspects of styles, neglecting styles like singing methods. moreover, they fail to conduct multi-level style control. 2) existing svs models often fail to generate singing voices rich in stylistic nuances for unseen singers. visinger 2 @xcite uses digital signal processing techniques to enhance synthesis quality. diffsinger @xcite employs a diffusion decoder to capture the intricacies of singing voices. however, these methods do not adequately incorporate style information into synthesis, leading to results that lack style variations in zero-shot tasks. to address these challenges, we introduce tc-singer, the first zero-shot svs model for style transfer across cross-lingual speech and singing styles, along with multi-level style control. tcsinger transfers and controls styles (like singing methods, emotion, rhythm, techniques, and pronunciation) from audio and text prompts to synthesize highquality singing voices. to model diverse styles (like singing methods, emotion, rhythm, technique, and pronunciation), we propose the clustering style encoder, which uses a clustering vector quantization (cvq) model to condense style information into a compact latent space, thus facilitating subsequent predictions, as well as enhance both training stability and reconstruction quality. for style transfer and control, we introduce the style and duration language model (s&d-lm). the s&d-lm incorporates a multi-task language module using audio and text prompts to concurrently predict style information and phoneme duration, thereby enhancing both. to generate singing voices rich in stylistic nuances, we introduce the style adaptive decoder, which employs a novel mel-style adaptive normalization method to refine mel-spectrograms with decoupled style information. our experimental results show that tcsinger outperforms other current best-performing baseline models in metrics including synthesis quality, singer similarity, and style controllability across various tasks, including zero-shot style transfer, multi-level style control, cross-lingual style transfer, and speech-to-singing (sts) style transfer. overall, our main contributions can be summarized as follows: ## tcsinger in this section, we first overview the proposed tc-singer. then, we introduce several critical components, including the clustering style encoder, the style adaptive decoder, and the style and duration language model (s&d-lm). finally, we elaborate on the training and inference procedures. ## conclusion in this paper, we introduce tcsinger, the first zeroshot svs model for style transfer across crosslingual speech and singing styles, along with multilevel style control. tcsinger transfers and controls styles (like singing methods, emotion, rhythm, technique, and pronunciation) from audio and text prompts to synthesize high-quality singing voices. the performance of our model is primarily enhanced through three key components: 1) the clustering style encoder that stably condenses style information into a compact latent space using a cvq model, thus facilitating subsequent predictions; 2) the style and duration language model (s&d-lm), which predicts style information and phoneme duration simultaneously, which benefits both; and 3) the style adaptive decoder that em-ploys a novel mel-style adaptive normalization method to generate enhanced details in singing voices. experimental results demonstrate that tc-singer surpasses baseline models in synthesis quality, singer similarity, and style controllability across zero-shot style transfer, multi-level style control, cross-lingual style transfer, and sts style transfer.