A lign STS : Speech-to-Singing Conversion via Cross-Modal Alignment (2023)

Abstract: The speech-to-singing (STS) voice conversion task aims to generate singing samples corresponding to speech recordings while facing a major challenge: the alignment between the target (singing) pitch contour and the source (speech) content is difficult to learn in a text-free situation. This paper proposes AlignSTS, an STS model based on explicit cross-modal alignment, which views speech variance such as pitch and content as different modalities. Inspired by the mechanism of how humans will sing the lyrics to the melody, AlignSTS: 1) adopts a novel rhythm adaptor to predict the target rhythm representation to bridge the modality gap between content and pitch, where the rhythm representation is computed in a simple yet effective way and is quantized into a discrete space; and 2) uses the predicted rhythm representation to re-align the content based on cross-attention and conducts a cross-modal fusion for re-synthesize. Extensive experiments show that AlignSTS achieves superior performance in terms of both objective and subjective metrics. Audio samples are available at https://alignsts.github.io .

## introduction speech-to-singing (sts) voice conversion @xcite @xcite aims to transfer speech samples into the corresponding singing samples, with the timbre identity and phoneme information unaltered. an sts system takes a speech sample and a target melody as conditions and then generates a high-quality singing sample following the target musical melody. speechto-singing research is important for human voice study and useful for practical applications such as computer-aided music production or musical entertainment. researchers have developed three major sts approaches: 1) model-based approaches @xcite use phone-score synchronization information to manually align the phonemes and the target musical notes with artificial control * * corresponding author models. 2) template-based sts @xcite @xcite ) requires an available high-quality reference vocal input, which will be aligned with the input speech for subsequent musical feature extractions. the alignment is the key part and is mostly based on dynamic time warping (dtw). 3) style transfer approach @xcite views sts as a style-transfer problem. this class of methods considers the specific properties that are transformed during conversion as "style". @xcite stretch the input speech to the same length as the target f0 contour and concatenate the latent features to fuse the asynchronous representations. @xcite serve as a continuation and extension of their prior work by leveraging boundary-equilibrium gan (began) @xcite . despite their recent success, however, the style information is complex and is composed of multiple entangled features in the time domain (the duration information and the temporal length) and the frequency domain (the pitch information). simply stretching the representations temporally or applying implicit self-attention can cause alignment problems. in a larger sense, human voice information (such as speech @xcite and singing @xcite ) is composed of several variance information, each controlling a specific sensory modality. plenty of arts focus on the decomposition and resynthesis of speech signal @xcite @xcite @xcite . they managed to roughly decompose speech signals into components like linguistic content, pitch, rhythm, timbre identity, etc. by manipulating any of these components, one can resynthesize a customized speech waveform. in the context of sts, the components that will be manipulated are 1) the frequency modality, namely the pitch information; and 2) the duration modality, or rhythm. the challenge remains, however, since the rhythm infor-mation can be indeterminate given only the content and the target pitch information. in this paper, we present a novel approach align-sts based on modality disentanglement and crossmodal alignment. to tackle the alignment problem, we introduce a novel rhythm representation to bridge the modality gap between content and pitch. a rhythm adaptor is designed to predict the target rhythm representation, which is used to guide the content features to perform the alignment and the cross-modal fusion. further, we explore speech-to-sing conversion in zero-shot scenarios, where we train the model with singing data in a self-supervised manner and test it on unseen speech data. we categorize alignsts as one of the style transfer approaches since we only need a source speech and a target pitch contour for conversion. experimental results demonstrate significant performance improvements over baseline models. the main contributions of this work are summarized as follows: ## alignsts in this section, we first define and formulate the problem of speech-to-singing (sts) voice conversion. we then present the information perturbation methods, the rhythm modality modeling, and the cross-modal fusion mechanism. finally, we introduce the overall architecture of alignsts and the training/inference procedure. ps re rhythm adaptor cross-modal aligner mel decoder rhythm embed e p e r e c timbre embed singing speech f0 vector quant. e p e c e r 4 1 1 2 1 embedding 4 1 1 2 1 cross entropy cross attention cross attention e c rhythm embed q k,v (a) overall (b) cross-modal aligner (c) rhythm adaptor rhythm index cross attention conv stack q k,v rhythm predictor k,v q wav2vec 2.0 ## conclusion in this work, we presented alignsts, a speechto-singing model based on modality disentanglement and cross-modal alignment. to achieve better voice quality and improve interpretability, we decomposed the input speech waveform into four variance information and proposed a novel cross-modal fusion mechanism. specifically, we designed a rhythm adaptor to adjust the rhythm representation to deal with the altered pitch modality, and a crossmodal aligner to re-align the content representation. finally, we conduct a cross-modal fusion to combine the different components together. experimental results demonstrated that align-sts achieved superior quality and naturalness compared with several baselines. ablation studies demonstrated that each design in alignsts was effective. extensive experiments showed the great potential of alignsts in zero-shot sts. we envisage that our work could serve as a basis for future sts studies.