Self-Supervised Singing Voice Pre-Training towards Speech-to-Singing Conversion (2024)

Abstract: Speech-to-singing voice conversion (STS) task always suffers from data scarcity, because it requires paired speech and singing data. Compounding this issue are the challenges of content-pitch alignment and the suboptimal quality of generated outputs, presenting significant hurdles in STS research. This paper presents SVPT, an STS approach boosted by a self-supervised singing voice pre-training model.We leverage spoken language model techniques to tackle the rhythm alignment problem and the in-context learning capability to achieve zero-shot conversion. We adopt discrete-unit random resampling and pitch corruption strategies, enabling training with unpaired singing data and thus mitigating the issue of data scarcity. SVPT also serves as an effective backbone for singing voice synthesis (SVS), offering insights into scaling up SVS models. Experimental results indicate that SVPT delivers notable improvements in both STS and SVS endeavors. Audio samples are available at https://speech2sing.github.io.

## introduction a speech-to-singing voice conversion (sts) system @xcite @xcite transforms the semantic content of a speech signal into a paired singing signal, conditioned on any form of pitch information, such as fine-grained f0 sequence @xcite . beyond being part of the development of music entertainment, works on sts provide insightful perspectives on bridging the gap between well-developed speech-language models and rudimentary singing voice modeling. although research in sts has achieved notable success recently, it continues to encounter several challenges. the major concern is the scarcity of paired speech-singing voice data. previous works rely on pre-made paired datasets @xcite , which have a much smaller quantity than singing voice corpora. the scarcity of data also makes sts models difficult to generalize or scale up. in addition, the non-autoregressive design in previous works exacerbates the misalignment issue. although @xcite tries to solve this problem by utilizing a roughly monotonic cross-attention mechanism, the attention map has a chance to degrade and corrupt when facing complex, long sentences. recent breakthroughs in large-scale generative spoken language models and audio discretization methods have opened up new frontiers in speech generation @xcite @xcite . to lighten the burden of directly modeling the target acoustic space, researchers split the models into two stages and map the prompts into an intermediate semantic latent space first, as illustrated in figure 1 . these achievements, however, have not significantly benefited the field of singing voice synthesis (svs) @xcite . because, unlike speech, the highly dynamic nature of singing voice and the scarcity of annotated data make it a challenge for language models (lms). either the frequency domain (pitch information) or temporal domain (rhythm component, or phoneme durations) of singing voices is highly unstable. luckily, these problems and the challenges of sts mentioned before have a chance to solve each other. 1 9819 sts models do not need textual annotations, while the in-context learning capability of spoken language models allows for an effective solution to the misalignment issue. as shown in figure 1 , stage 2, translating high-level semantic representations into low-level acoustic representations, requires no annotations or transcriptions. similarly, @xcite disentangles the input speech signal into corrupted semantic-related content representations and transforms them into mel-spectrograms, conditioned on pitch information. therefore, stage 2 in a spoken language model is naturally suited to sts tasks. we claim that with proper regulations and enhancements, spoken language models are self-supervised singing voice learners, which make use of abundant unannotated data and bridge the gap between speech and singing voice. in this paper, we propose svpt, an sts approach boosted by a self-supervised singing voice pre-training model, which further improves singing voice synthesis. we construct a multiscale transformer with a decoder-only architecture @xcite , specializing in long-sequence modeling like discrete audio tokens. special regulations and perturbations, such as discrete-unit random resampling and expanded-range reference prompting, are introduced to generalize and stabilize the model. essentially, the proposed model translates the corrupted semantic tokens into the target acoustic tokens autoregressively, conditioned on pitch information and sampled reference prompt tokens. the benefits are three-fold: (a) the models extracting the semantic and acoustic tokens are pretrained and finetuned in a self-supervised fashion, making this procedure fully annotation-free; (b) the semantic and acoustic tokens can be derived from singing voice samples solely, where the former are corrupted and disentangled to have a closer distribution distance to speech signals; (c) with an external text-to-semantic translator, the sts model can be upgraded to a high-quality zero-shot singing voice synthesizer. our contributions are summarized as follows: ## conclusion this paper proposed svpt, the first sts approach boosted by a self-supervised singing voice pretraining model, which further improved singing voice synthesis. we adopted discrete-unit random resampling and pitch corruption strategies to stabilize and generalize the training of the spoken language model. with an external text-to-semantic translator, svpt served as an effective backbone for svs and provided an opportunity for svs models to scale up. experimental results revealed notable performances in both sts and svs tasks.