Improving C hinese Pop Song and Hokkien Gezi Opera Singing Voice Synthesis by Enhancing Local Modeling (2023)

Abstract: Singing Voice Synthesis (SVS) strives to synthesize pleasing vocals based on music scores and lyrics. The current acoustic models based on Transformer usually process the entire sequence globally and use a simple L1 loss. However, this approach overlooks the significance of local modeling within the sequence and the local optimization of the hard-to-synthesize parts in the predicted mel-spectrogram. Consequently, the synthesized audio exhibits local incongruities ( e.g. , local pronunciation jitter or local noise). To address this problem, we propose two methods to enhance local modeling in the acoustic model. First, we devise a nearest neighbor local attention, where each phoneme token focuses only on the adjacent phoneme tokens located before and after it. Second, we propose a phoneme-level local adaptive weights loss function that enables the model to focus more on the hard-to-synthesize parts of the mel-spectrogram. We have verified the universality of our methods on public Chinese pop song and Hokkien Gezi Opera datasets. Extensive experiments have demonstrated the effectiveness of our methods, resulting in significant improvements in both objective and subjective evaluations when compared to the strong baselines. Our code and demonstration samples are available at https://github.com/baipeng1/SVSELM.

## introduction singing voice synthesis (svs) converts the lyrics into natural and humanlike voice audio according to the music scores @xcite 1 . due to its promising application in fields such as virtual singer and music education, svs has attracted the attention of a large number of researchers recently @xcite @xcite @xcite . svs systems generally consist of an acoustic model and * equal contribution. a vocoder. the acoustic model converts music scores and lyrics into acoustic features (e.g., melspectrogram), and the vocoder synthesizes audio waveform from acoustic features @xcite . recently, transformer @xcite ) has been widely used in sequence modeling tasks. the acoustic models based on transformer have showed great performance, including fft-npss @xcite , xiaoicesing @xcite , deepsinger @xcite , fft-singer @xcite . however, these models still exhibits local incongruity in the synthesized audio, which is characterized by local pronunciation jitter or noise. local incongruity will bring negative experiences to listeners, so this is a problem that urgently needs to be solved. @xcite also focus on local incongruity problem. in order to improve the accuracy of the local pronunciation, they added a postnet to the model and used adversarial training methods, where the voicing-aware discriminator was used to capture the harmonic features of vocal segments and the noise components of silent segments. unlike their approaches, we abstain from employing post processing networks or adversarial training methods. instead, we address this problem from the perspective of enhancing local attention and refining loss function. some studies @xcite @xcite discovered that incorporating additional local attention can enhance model performance in text to speech (tts) and automatic speech recognition (asr) tasks. in addition, some studies @xcite @xcite used the local focus loss function in the image reconstruction task to improve the reconstruction effect. inspired by the above works, in order to address the problem of local incongruity in acoustic models based on transformer, we propose two methods to enhance local modeling. first, the attention mechanism in these acoustic models is the global contextual the main contributions of this paper can be summarized as follows: ## methods in section 2.1, we first introduce an overview of our model. we then introduce the nearest neighbor local attention method in section 2.2, and finally introduce the local adaptive weights loss method in section 2.3. ## results this section shows the results of the experiments. section 4.1 is the main comparison and analysis of the overall objective and subjective results. section 4.2 is the experiments about the best number selection for the previous and next masked phoneme tokens. section 4.3 explores how our methods can flexibly combine with other models, such as diffsinger. section 4.4 is a case study, which visually demonstrates the effectiveness and universality of our proposed methods. ## conclusion in the chinese pop song and hokkien gezi opera singing voice synthesis tasks, we propose two local modeling enhancement methods in acoustic model based on transformer to improve the quality of the predicted mel-spectrogram. one method is to enhance local attention for each phoneme token in the decoder and fuse local attention representation with the original global self-attention representation. another method involves employing a novel phoneme-level local adaptive weights loss to optimize the hard-to-synthesize parts of the predicted mel-spectrogram. we conduct extensive experiments on the chinese pop song and hokkien gezi opera datasets, and both objective and subjective evaluation metrics show the effectiveness and universality of our methods in enhancing local modeling for mel-spectrogram prediction. our two methods are simple and practical, and can be flexibly incorporated into acoustic models based on transformer or conformer. in summary, our methods can greatly alleviate the local inconsistency problem in svs tasks and improve the quality of synthesized audio. we are moving towards a better solution to completely solve this problem.