Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt (2024)

Abstract: Recent singing-voice-synthesis (SVS) methods have achieved remarkable audio quality and naturalness, yet they lack the capability to control the style attributes of the synthesized singing explicitly. We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language. We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation that enables text-conditioned vocal range control while keeping melodic accuracy. Furthermore, we explore various experiment settings, including different types of text representations, text encoder fine-tuning, and introducing speech data to alleviate data scarcity, aiming to facilitate further research. Experiments show that our model achieves favorable controlling ability and audio quality. Audio samples are available at http://prompt-singer.github.io .

## introduction singing-voice-synthesis (svs) systems @xcite @xcite @xcite , which aim to generate high-fidelity singing voices given lyrics and pitch notes, have made significant advancements in improving audio quality and naturalness in recent years, facilitating music composition and development of entertainment industries. however, it hasn't been fully studied to control the style attributes of synthesized singing, such as speaker timbre, vocal range and energy. despite that some works use fixed speaker ids @xcite or reference speech/singing segments @xcite to provide information on singer identity or other style attributes, these mechanisms are not user-friendly and lack the ability to control specific acoustic attributes explicitly. an ideal approach to controlling the style of generated singing voices is to use natural language instructions as style prompts, as it can not only achieve precise control over specific attributes with certain descriptions, but also simplify user interaction, which may bring convenience to nonprofessional users such as musicians and video creators. however, applying natural language style prompts in singing-voice-synthesis faces several challenges: in this paper, we propose prompt-singer, the first controllable svs model with natural language prompts to control the singer gender, vocal range and volume. considering the outstanding performance of recent spoken llms @xcite @xcite in terms of generation and in-context learning capabilities, we adopt a decoder-only transformer with a multi-scale hierarchy for conditional generation of discrete codec units of the singing, together with a unit vocoder for waveform reconstruction. to address the challenges mentioned above, we 1) design a decoupled pitch representation with a vocal range factor and a speaker-independent melody sequence, enabling voice range controlling while maintaining melodic accuracy; 2) investigate various text encoders for prompt encoding, as well as fine-tuning the encoders to seek the optimal textual representation for this task; 3) introduce speech data to alleviate data scarcity, and evaluate the model performance under different levels of low-resource singing data combined with speech data. experiments show that our method achieves favorable style controlling accuracy on the three attributes, while keeping good audio quality and melodic accuracy. our contributions are summarized as follows: ## prompt generation and fetching our goal is to control the singer gender, vocal range and volume in singing-voice-synthesis with natural language prompts. since there is no available dataset for this task, we utilize normal svs datasets and design a method for generating a prompt sentence for each data item. we introduce this process in this section. considering the high cost of manual annotation, we utilize a large language model (gpt 3.5 turbo) singing data fetching label gender: female volume: medium pitch: high female: woman, lady, girl, lass … medium volume: normal, common, moderate… random drop gender: female volume: medium fetching synonyms fetching sentence template sentence templates: i'd like to listen to a vivid song with a [gender] vocalist, and it should have a [volume] voice, i want a song by a … prompt sentence assembling i'd like to listen to a vivid song with a girl vocalist, and it should have a moderate voice. to generate prompt sentences. the prompt generation mainly consists of 3 stages: 1) attribute categorization; 2) keyword and sentence template generation and 3) prompt sentence assembling. figure @xcite and (b) demonstrate the process of the first two stages. initially, we categorize the audio based on different attributes. the two gender categories, male and female, are pre-annotated in the datasets. for volume, we build three categories of "low", "medium", and "high", indicating the amplitude root mean square (rms) ranges of [0.02, 0.04], [0.07, 0.10] and [0.16, 0.20], respectively. additionally, we can rescale audio into different ranges dynamically during training. for vocal range, we set two categories of "high" and "low", and use the average f0 of the voiced part as the criterion for classification, with the threshold being 125 hz for male singers and 305 hz for female singers. after categorization, we use the llm to generate a set of 4-7 synonyms for each category as the keywords. we further utilize the llm to generate prompt sentence templates for each single attribute, where each template contains a placeholder to be replaced with the keywords (such as generate a song by a [gender] singer). we also generate a small number of prompt sentences targeting specific categories (such as could you synthesize a song that's as powerful as a thunderstorm? for large volume). we obtain approximately 50 sentence templates for each attribute after manual selection. these single-attribute templates can be further combined to create multi-attribute templates by prompting the llm. we provide sample sentence templates and keywords in appendix a. the prompt sentence assembling stage takes place dynamically during training. figure 1(c ) illustrates the pipeline of fetching a prompt sentence. we first obtain the pre-annotated labels for the data item, and in order to make the model adaptable to prompts with varying numbers of attributes, one or two labels are randomly dropped with probabilities p 1 and p 2 . we then randomly fetch a keyword and a sentence template from the pre-generated sets, and replace the placeholder with the keyword to get the final prompt sentence. note that we do not control vocal range independently in the absence of gender, as its boundary is different for male and female. we use pre-generated specific prompts for each sample in the evaluation for fair comparison. ## prompt-singer in this section, we introduce the model design of prompt-singer. the overall architecture of our model is illustrated in figure 2 @xcite . it is primarily composed of two sub-modules: 1) the multi-scale transformer, which generates discrete acoustic units conditioned on inputs of natural language prompt, lyrics with duration, and pitch information; and 2) the unit vocoder, which maps the generated acoustic units to an audio waveform. in the following subsections, we introduce the input and output representations of the model in section 4.1 to 4.3, model architecture in detail in section 4.5 and 4.6, together with our method for data scarcity alleviation in section 4.4. ## conclusion in this paper, we propose prompt-singer, the first singing-voice-synthesis method with the ability of style control using natural language prompts. we adopt a multi-scale decoder-only transformer for generating acoustic units of singing, followed by a unit-vocoder for audio reconstruction. we design a decoupled pitch representation for vocal range modification with an accurate melody kept. furthermore, we investigate various experiment settings, including different text representations, finetuning the text encoders, and using speech data to boost performance in low-resource scenarios. in future works, we plan to introduce more style attributes in controllable svs, such as emotion, rhythm and more detailed singer information. we hope our work will facilitate the development of the svs community. despite that our model achieves remarkable controlling capability and audio quality on prompt singing-voice-synthesis, it still has two major limitations: 1) due to the simplicity and inflexibility of our existing prompt generation pipeline, the generated prompt texts may suffer from distributional bias, manifested mainly as grammatical errors, unnatural expressions, and restrictions in expressive capacity and diversity. we suggest that a potential solution is to pass the assembled prompt sentences through the llm once more for refinement and synonymous sentence generation to improve accuracy and expressiveness. 2) due to the utilization of large-scale models (including the text encoders and the transformer backbone) along with an autoregressive generation paradigm, our model entails relatively high computational overhead, resulting in considerable inference latency. we discuss the relationship between inference latency and the length of the generated audio in appendix e. besides, misuse of our model for singing voice generation may lead to copyright issues. we will add some constraints to guarantee people who use our code or pre-trained model will not use the model in illegal cases.