STARS : A Unified Framework for Singing Transcription, Alignment, and Refined Style Annotation (2025)

Abstract: Recent breakthroughs in singing voice synthesis (SVS) have heightened the demand for high-quality annotated datasets, yet manual annotation remains prohibitively labor-intensive and resource-intensive. Existing automatic singing annotation (ASA) methods, however, primarily tackle isolated aspects of the annotation pipeline. To address this fundamental challenge, we present STARS, which is, to our knowledge, the first unified framework that simultaneously addresses singing transcription, alignment, and refined style annotation. Our framework delivers comprehensive multi-level annotations encompassing: (1) precise phoneme-audio alignment, (2) robust note transcription and temporal localization, (3) expressive vocal technique identification, and (4) global stylistic characterization including emotion and pace. The proposed architecture employs hierarchical acoustic feature processing across frame, word, phoneme, note, and sentence levels. The novel non-autoregressive local acoustic encoders enable structured hierarchical representation learning. Experimental validation confirms the framework’s superior performance across multiple evaluation dimensions compared to existing annotation approaches. Furthermore, applications in SVS training demonstrate that models utilizing STARS-annotated data achieve significantly enhanced perceptual naturalness and precise style control. This work not only overcomes critical scalability challenges in the creation of singing datasets but also pioneers new methodologies for controllable singing voice synthesis.

## introduction automatic singing annotation (asa) constitutes the computational process of extracting key vo- † equal contribution * corresponding author cal features from singing recordings, encompassing phonetic transcriptions (phoneme alignment), midi note parameters (pitch/duration), and stylistic attributes (emotion/technique). as the cornerstone of modern singing voice synthesis (svs) systems, asa provides fine-grained, multi-level annotated data for training expressive and controllable singing synthesis models. while recent breakthroughs in generative models and controllable svs frameworks @xcite have dramatically improved the quality of generated singing voices, they have also paradoxically exposed a critical bottleneck: the scarcity of high-quality annotated singing corpora. traditional annotation workflows require laborintensive manual processing by audio engineers and musicians, making large-scale dataset creation both costly and time-consuming. while some open-source singing datasets such as opencpop @xcite and vocalset @xcite have attempted to alleviate this burden, their annotations are limited to basic phonetic or vocal technique information. recent datasets like gtsinger @xcite have made significant progress by incorporating a wider range of annotations-from basic phoneme and note annotations to various singing techniques and global styles. however, the volume of data remains insufficient compared to the scale of speech corpora. modern svs systems require multi-level annotation precision across four key dimensions: to overcome these challenges, we propose stars, a unified framework for multi-level singing voice annotation that streamlines the entire process. stars offers three key innovations: (1) a multi-level architecture for extracting singing information at various granularities, covering frame, word, phoneme, note, and sentence levels; (2) a local acoustic encoder that works together with a cmu encoder employing a u-net architecture with conformer blocks and freqmoe, and with a vector quantization, to extract acoustic features at multiple hierarchical levels; and (3) a multitask automated annotation pipeline that sequentially predicts phoneme boundaries, note boundaries, pitch, phone-level techniques, and global stylistic attributes. we demonstrate the effectiveness of stars through comprehensive evaluations across multiple asa tasks. our framework achieves superior performance in phoneme alignment accuracy and note prediction precision. when applied to svs model training, stars-annotated data yields significant improvements in synthesized voice naturalness and style control accuracy. ## conclusion in this paper, we introduce stars, the first unified framework for singing transcription, alignment, and refined style annotation. we construct a multi-level framework that efficiently extracts audio features at five granularities-frame, word, phone, note, and sentence-using a hierarchical acoustic encoder. our approach enables a complete automatic singing annotation pipeline, sequentially performing singing lyric alignment, note transcription and alignment, phone-level technique predic-tion, and global style prediction. experimental results demonstrate that our model achieves high performance across all sub-tasks, and the annotated outputs are further validated in a singing synthesis task, confirming the effectiveness of our approach.