RMSS inger: Realistic-Music-Score based Singing Voice Synthesis (2023)

Abstract: We are interested in a challenging task, Realistic-Music-Score based Singing Voice Synthesis (RMS-SVS). RMS-SVS aims to generate high-quality singing voices given realistic music scores with different note types (grace, slur, rest, etc.). Though significant progress has been achieved, recent singing voice synthesis (SVS) methods are limited to fine-grained music scores, which require a complicated data collection pipeline with time-consuming manual annotation to align music notes with phonemes. % Furthermore, existing approaches cannot synthesize rhythmic singing voices given realistic music scores due to the domain gap between fine-grained music scores and realistic music scores. Furthermore, these manual annotation destroys the regularity of note durations in music scores, making fine-grained music scores inconvenient for composing. To tackle these challenges, we propose RMSSinger, the first RMS-SVS method, which takes realistic music scores as input, eliminating most of the tedious manual annotation and avoiding the aforementioned inconvenience. Note that music scores are based on words rather than phonemes, in RMSSinger, we introduce word-level modeling to avoid the time-consuming phoneme duration annotation and the complicated phoneme-level mel-note alignment. Furthermore, we propose the first diffusion-based pitch modeling method, which ameliorates the naturalness of existing pitch-modeling methods. To achieve these, we collect a new dataset containing realistic music scores and singing voices according to these realistic music scores from professional singers. Extensive experiments on the dataset demonstrate the effectiveness of our methods. Audio samples are available at https://rmssinger.github.io/ .

## introduction singing voice synthesis (svs) aims to generate high-quality singing given music scores (lyrics, figure 1 : difference between fine-grained and realistic music scores. manual adjustment destroys the regularity of note durations, making fine-grained music scores "crushed" and inconvenient for composing. note pitches, and note durations), and has attracted increasing academic and industrial attention. svs is extensively required in both professional music composing and entertainment industries in real life @xcite . though significant progress has been achieved, recent svs methods @xcite @xcite @xcite cannot utilize realistic music scores from composers or websites but require finegrained music scores. fine-grained music scores are obtained through a complicated data collection pipeline, which can be mainly divided into three major steps @xcite : 1) phoneme annotation step, where the duration of each phoneme is first extracted from singing through montreal forced aligner foot_0 and then further manually annotated to acquire more accurate phoneme boundaries. 2) note annotation step, where preliminary notes are either created by logic pro @xcite or collected through the word-level average of extracted f0 @xcite and then note durations are manually adjusted to the boundaries of vowel phonemes. 3) silence annotation step, where the silence part is annotated as silence (sp) or aspirate (ap). these steps, especially the first and second steps, require arduous and professional manual annotation @xcite , which hinders large-scale svs data collection. furthermore, since the manual adjustment in the second step destroys the regularity of note durations, fine-grained music scores have a noteworthy difference from realistic music scores (see figure 1 ), which inhibits human composers from employing svs methods for composing. the primary rationale for adopting such a timeconsuming and laborious data collection pipeline is twofold: 1) existing methods require phonemelevel hard-alignment for duration training. due to the difficulty in determining phoneme boundaries @xcite , complex manual annotation (phoneme annotation step) is necessary in order to prevent the negative effects of incorrect alignment on model training. 2) existing methods require a pre-defined phoneme-level mel-note alignment for training and inference. since one vowel phoneme may correspond to multiple notes @xcite , existing methods directly repeat this phoneme to conform with notes, which requires note boundaries to be aligned with the boundary of each vowel phoneme. however, even professional singers can hardly sing fully conformed to the music score @xcite , so the note annotation step has to be performed. to tackle these challenges, we introduce realistic-music-score singer (rmssinger), the first rms-svs method, which utilizes realistic music scores with different note types (grace, slur, rest, etc.,) for training and inference, alleviating most manual annotations. to alleviate the tedious annotation in the phoneme annotation step, we propose word-level positional attention with wordlevel hard-alignment and positional attention to avoid the difficulty of determining exact phoneme boundaries. to avoid the note annotation step, we propose the word-level learned gaussian upsampler to learn the word-level mel-note alignment in training and avoid the phoneme-level mel-note alignment. furthermore, existing methods mainly adopt simple l1 or l2 loss for pitch modeling, which results in the degradation of expressiveness. to achieve expressive pitch prediction, we propose the first diffusion-based pitch generation method. due to the existence of both continuous parts (f0) and categorical parts (uv) in pitch contours, we propose the pitch diffusion model (p-ddpm), which models categorical uv and continuous f0 in a single model. extensive experiments on our collected datasets demonstrate the efficiency of our proposed word-level framework (word-level positional attention and word-level learned gaussian upsampler) and p-ddpm. the main contributions of this work are summarized as follows: ## related works singing voice synthesis (svs) aims to generate high-quality singing conditioned on given music scores. with the development of deep learning, svs has achieved great progress in the network structure and the singing corpus construction. xiaoicesing @xcite adopts the non-autoregressive acoustic model inspired by fastspeech @xcite . bytesing @xcite is designed based on the autoregressive tacotron-like @xcite architecture. deepsinger @xcite builds a singing corpus by mining singing data from websites and proposes the singing model based on the feed-forward transformer @xcite . more recently, opencpop @xcite publish a single-singer chinese song corpus with manually-annotated fine-grained music scores and propose a conformer-based @xcite svs method. wesinger @xcite adopts a transformer-like acoustic model and an lpcnet neural vocoder. visinger @xcite employs the vits @xcite architecture for end-to-end svs and introduces an f0 predictor to guide the prior network. diff-singer @xcite introduces the diffusionbased @xcite decoder for the highquality mel-spectrogram generation and proposed the shallow diffusion mechanism for faster inference. m4singer @xcite further publishes a multi-style, multi-singer chinese song corpus with manually-annotated fine-grained music scores. ## diffusion models diffusion models @xcite @xcite are a paradigm of generative methods that aim to approximate the end-point distribution (target distribution) of a markow chain and have achieved impressive results in benchmark generative tasks @xcite . diffusion models consist of two processes: diffusion process the diffusion process gradually perturbs data x 0 ∼ q(x 0 ) to pure noise with a markov chain according to the variance schedule β 1 , ..., β t : reverse process the reverse process gradually denoises the latent variable x t ∼ p(x t ) to the corresponding real data sample x 0 : with different perturbation transition q(x t |x t-1 ) used, different diffusion models are defined: gaussian diffusion: gaussian diffusion @xcite is utilized in continuous data domains. gaussian diffusion adopts the gaussian noise for perturbation: with the parameterization introduced in @xcite , equation 3 can be further simplified and finally optimized with: and the neural network is trained to approximate x0 from noisy sample x t given timestep t. though widely utilized in many data domains, diffusion models have never been utilized for pitch modeling. furthermore, due to the existence of continuous f0 parts and discrete uv parts in pitch contours @xcite , neither gaussian diffusion nor multinomial diffusion alone can deal with pitch modeling. in this paper, we propose the first diffusion-based pitch modeling (p-ddpm), which incorporates gaussian diffusion and multinomial diffusion in a single model and achieves better pitch modeling. ## conclusion in this paper, we propose rmssinger, the first realistic-music-score-based singing voice synthesis (rms-svs) method, which utilizes the word-level modeling framework to avoid most tedious manual annotations. to achieve better pitch modeling, we propose the first diffusion-based pitch modeling method (p-ddpm), which incorporates the gaussian diffusion and multinomial diffusion in a single model. extensive experiments conducted on our collected dataset demonstrate the feasibility of our method for rms-svs and the superiority of our proposed p-ddpm. @xcite 2022b. wesinger: data-augmented singing voice synthesis with auxiliary losses. arxiv preprint arxiv:2203.10750.