Robust Singing Voice Transcription Serves Synthesis (2024)

Abstract: Note-level Automatic Singing Voice Transcription (AST) converts singing recordings into note sequences, facilitating the automatic annotation of singing datasets for Singing Voice Synthesis (SVS) applications. Current AST methods, however, struggle with accuracy and robustness when used for practical annotation. This paper presents ROSVOT, the first robust AST model that serves SVS, incorporating a multi-scale framework that effectively captures coarse-grained note information and ensures fine-grained frame-level segmentation, coupled with an attention-based pitch decoder for reliable pitch prediction. We also established a comprehensive annotation-and-training pipeline for SVS to test the model in real-world settings. Experimental findings reveal that the proposed model achieves state-of-the-art transcription accuracy with either clean or noisy inputs. Moreover, when trained on enlarged, automatically annotated datasets, the SVS model outperforms its baseline, affirming the capability for practical application. Audio samples are available at https://rosvot.github.io. Codes can be found at https://github.com/RickyL-2000/ROSVOT.

## introduction note-level automatic singing voice transcription (ast) refers to converting a singing voice recording into a sequence of note events, including note pitches, onsets, and offsets @xcite @xcite . as part of the music information retrieval (mir) task, ast is widely used in professional music production and post-production tuning. with the recent advancements of singing voice synthesis (svs) @xcite @xcite , there is a growing demand for annotated data, while ast methods just demonstrate the potential for automatic annotation. ## experiments in this section, we begin by showcasing experiments on ast tasks, followed by simulations and comparisons of a comprehensive annotation-andtraining pipeline for an svs task. we also investigated the model's performance in low-resource scenarios; however, due to space limitations, this part is included in appendix g. ## conclusion in this paper, we introduced rosvot, the first robust ast model that ultimately serves svs. we leveraged a multi-scale architecture to achieve a balance between coarse-grained note modeling and fine-grained segmentation. an attention-based decoder with dynamic weight was devised for pitch regression. additionally, we established a comprehensive pipeline for svs training. experimental results revealed that our model achieved the best performance under either clean or noisy environments. annotating and incorporating larger datasets improved the svs model's performance, indicating the capability of practical annotation of rosvot.