Learning the Beauty in Songs: Neural Singing Voice Beautifier (2022)

Abstract: We are interested in a novel task, singing voice beautification (SVB). Given the singing voice of an amateur singer, SVB aims to improve the intonation and vocal tone of the voice, while keeping the content and vocal timbre. Current automatic pitch correction techniques are immature, and most of them are restricted to intonation but ignore the overall aesthetic quality. Hence, we introduce Neural Singing Voice Beautifier (NSVB), the first generative model to solve the SVB task, which adopts a conditional variational autoencoder as the backbone and learns the latent representations of vocal tone. In NSVB, we propose a novel time-warping approach for pitch correction: Shape-Aware Dynamic Time Warping (SADTW), which ameliorates the robustness of existing time-warping approaches, to synchronize the amateur recording with the template pitch curve. Furthermore, we propose a latent-mapping algorithm in the latent space to convert the amateur vocal tone to the professional one. To achieve this, we also propose a new dataset containing parallel singing recordings of both amateur and professional versions. Extensive experiments on both Chinese and English songs demonstrate the effectiveness of our methods in terms of both objective and subjective metrics. Audio samples are available at https://neuralsvb.github.io . Codes: https://github.com/MoonInTheRiver/NeuralSVB .

## introduction the major successes of the artificial intelligent singing voice research are primarily in singing voice synthesis (svs) @xcite @xcite @xcite and singing voice conversion (svc) @xcite @xcite . however, the singing voice beautification (svb) remains an important and challenging endeavor for researchers. svb aims to improve the * corresponding author intonation foot_0 and the vocal tone of the voice, while keeping the content and vocal timbre foot_1 . svb is extensively required both in the professional recording studios and the entertainment industries in our daily life, since it is impractical to record flawless singing audio. nowadays in real-life scenarios, svb is usually performed by professional sound engineers with adequate domain knowledge, who manipulate commercial vocal correction tools such as melodyne 3 and autotune foot_3 @xcite . most current automatic pitch correction works are shown to be an attractive alternative, but they may 1) show weak alignment accuracy @xcite or pitch accuracy @xcite ; 2) cause the tuned recording and the reference recording to be homogeneous in singing style @xcite . besides, they typically focus on the intonation but ignore the overall aesthetic quality (audio quality and vocal tone) @xcite . to tackle these challenges, we introduce neural singing voice beautifier (nsvb), the first generative model to solve the svb task, which adopts a conditional variational autoencoder (cvae) @xcite as the backbone to generate high-quality audio and learns the latent representation of vocal tone. in nsvb, we dichotomize the svb task into pitch correction and vocal tone improvement: 1) to correct the intonation, a straightforward method is aligning the amateur recording with the template pitch curve, and then putting them together to resynthesize a new singing sample. previous works @xcite implemented this by figuring out the alignment through dynamic time warping (dtw) @xcite or canonical time warping (ctw) @xcite . we propose a novel shape-aware dtw algorithm, which ameliorates the robustness of existing time-warping approaches by considering the shape of the pitch curve rather than low-level features when calculating the optimal alignment path. 2) to improve the vocal tone, we propose a latentmapping algorithm in the latent space, which converts the latent variables of the amateur vocal tone to those of the professional ones. this process is optimized by maximizing the log-likelihood of the converted latent variables. to retain the vocal timbre during the vocal tone mapping, we also propose a new dataset named popbutfy containing parallel singing recordings of both amateur and professional versions. besides, thanks to the autoencoder structure, nsvb inherently supports semisupervised learning, where the additional unpaired, unlabeled foot_4 singing data could be leveraged to facilitate the learning of the latent representations. extensive experiments on both chinese and english songs show that nsvb outperforms previous methods by a notable margin, and each component in nsvb is effective, in terms of both objective and subjective metrics. the main contributions of this work are summarized as follows: ## methdology in this section, we describe the overview of nsvb, which is shown in figure 1 . at stage 1 in the figure, we reconstruct the input mel-spectrogram through the cvae backbone (section 3.1) based on the pitch, content and vocal timbre conditions extracted from the input by the pitch encoder, content encoder and timbre encoder, and optimize the cvae by maximizing evidence lower bound and adversarial learning. at stage 2/inference in the figure, firstly we infer the latent variable z a based on the amateur conditions; secondly we prepare the amateur content vectors aligned with the professional pitch by sadtw algorithm (section 3.2); thirdly we map z a to z p by the latent-mapping algorithm (section 3.3); finally, we mix the professional pitch, the aligned amateur content vectors, and the amateur vocal timbre to obtain a new condition, which is leveraged along with the mapped z p by the decoder of cvae to generate a new beautified mel-spectrogram. the training/inference details and model structure of each component in nsvb are described in section 3.4 and section 3.5.