Text-to-Song: Towards Controllable Music Generation Incorporating Vocal and Accompaniment (2024)

Abstract: A song is a combination of singing voice and accompaniment. However, existing works focus on singing voice synthesis and music generation independently. Little attention was paid to exploring song synthesis. In this work, we propose a novel task called Text-to-Song synthesis which incorporates both vocal and accompaniment generation. We develop Melodist, a two-stage text-to-song method that consists of singing voice synthesis (SVS) and vocal-to-accompaniment (V2A) synthesis. Melodist leverages tri-tower contrastive pretraining to learn more effective text representation for controllable V2A synthesis. A Chinese song dataset mined from a music website is built to alleviate data scarcity for our research. The evaluation results on our dataset demonstrate that Melodist can synthesize songs with comparable quality and style consistency. Audio samples can be found in https://text2songMelodist.github.io/Sample/.

## introduction songs, as intricate musical compositions, have always enjoyed the greatest popularity among music enthusiasts. it inspires the pursuit of song synthesis by leveraging machine learning and artificial intelligence algorithms. it makes sense to generate a song conditioned on text modality (music score, natural language prompt, etc.). however, there is little exploratory research on text-to-song synthesis to our knowledge. there are two related tasks. the first is singing voice synthesis, which converts the music score @xcite to the singing voice. existing svs models have achieved remarkable achievement regarding quality @xcite @xcite and zero-shot ability @xcite figure 1 : the comparison of three tasks: singing voice synthesis, accompaniment generation and text-to-song. in this work, we investigate on the relationship between vocal and accompaniment for text-to-song synthesis. @xcite but they can only generate vocals. another similar task is music accompaniment generation @xcite , which usually aims at generating multi-track sequences in the symbolic domain or directly generating music waveform from text descriptions @xcite . as presented in figure 1 , there are similarities among these three tasks, while notable distinctions exist. the accompaniments are often removed in data preprocessing to train an svs model. existing music generation models do not take vocals into account as the condition. further exploration of text-to-song is inhibited. neither serves as the suitable prior. to address this limitation, we propose a novel generative task, text-to-song, which converts the music score (lyrics, notes, and duration) to the song, that is, singing voice with accompaniment. however, a text-to-song model is facing several challenges: in this paper, we propose melodist, the first textto-song model to generate music incorporating vocal and accompaniment from music score. to overcome the challenges mentioned above, we adopt several techniques: 1) based on the human perception that the accompaniment complements the vocal melody, providing harmonic and rhythmic structure to enhance the overall musical expression, we introduce a two-stage text-to-song synthesis. specifically, melodist generates singing voice from the music score in stage 1, then generates accompaniment given vocal in stage 2. finally, we mix the outputs of two stages to obtain the song. it releases the burden of our model to a large extent; 2) we utilize the attribute tags (mood, instruments, style, etc.) of each song segment and construct natural language prompts to guide the synthesis of the accompaniment. we further apply the tri-tower contrastive learning framework to extract better text representations; 3) we crawled some songs and the corresponding lyrics and tags related to attributes from music websites. we evaluate our model under different settings and the results demonstrate that melodist can synthesize songs with comparable quality under the control of natural language prompts. the main contributions of our work can be summarized as follows: ## two-stage text-to-song synthesis in this section, we first present a formal definition of text-to-song synthesis task. then we will give an overview of the proposed model melodist. finally, we will elaborate on the approaches we adopt for controllable two-stage text-to-song synthesis. ## 6251 figure 3 : the architecture of the tri-tower contrastive framework. z p , z v , z a refer to the representation extracted by the text encoder, the vocal encoder and the accompaniment encoder, respectively. we use different shapes to represent different triples, while color is used to distinguish the kinds of inputs. embeddings of the same triplet are pulled closer, while those of different objects are pushed away in the joint embedding space. ## tri-tower contrastive pre-training we introduce a tri-tower training scheme with contrastive loss that jointly embeds text, vocal, and accompaniment into an aligned space. as presented in figure 3 , it consists of three separate encoders: text encoder f p (•), vocal encoder f v (•), and accompaniment encoderf a (•), each followed by a pooling and linear layer. parallel text prompt, vocal, and accompaniment make up each triplet of a mini-batch (x p , x v , x a ) and they are passed through the respective encoder. the text encoder f p (•): a n -→ r d p converts a tokenized text sequence of length n over vocabulary a to the text embedding of dimension d. the vocal encoder and the accompaniment encoder f v (•), f a (•): r f ×t -→ r d v encode log mel spectrograms of the vocal and accompaniment respectively, which f refers to the number of mel channels and t refers to the number of frames. a linear layer is appended in each branch to project the representations into a l 2 -normalized embedding space. when considering two-tower contrastive learning, two encoders of different modalities are jointly trained to maximize the similarity between n positive pairs while minimizing the similarity for n ×(n -1) negative pairs. we adopt the multi-modal version of infonce loss @xcite . taking pair (text, vocal) as an example, the loss can be formulated as follows: where τ is a temperature parameter. to extend it into tri-tower contrastive loss, we simply calculate the contrastive loss over pairs of the representations in a triplet (text, vocal, accompaniment) that: to verify the effectiveness of the tri-tower contrastive pre-training framework, we also compare 5 6252 it with clap on two related cross-modal retrieval tasks: text-vocal retrieval and text-accompaniment retrieval. we report the experimental results in section 5.4.3, which indicates that including both vocal and accompaniment helps the model learn to ground more attribute-related song concepts. ## conclusion in this paper, we introduce a new task called textto-song, which incorporates singing voice and accompaniment synthesis from music score. we propose melodist, the first text-to-song model with a two-stage generation scheme. natural language prompts serve as the condition to control accompaniment generation. melodist leverage a tri-tower contrastive pre-training framework to align the prompt with its vocal and accompaniment. we have collected a mandarin song dataset from the music website and leveraged some open-source song and singing datasets to alleviate the data scarcity. we have conducted a series of comprehensive evaluations and the results indicate that melodist outperforms baselines with comparable audio quality, temporal correspondence, and consistency with text concept. we provide extensive experiments to demonstrate the effectiveness of the tri-tower contrastive learning framework as well as the impact of different data combinations and data augment strategies. in the future, we will focus on improving the audio quality and vocal accompaniment harmonization.