Lyrics Transcription in Western Classical Music with Whisper: A Case Study on Schubert’s Winterreise (2024)

Abstract: Automatic Lyrics Transcription (ALT) aims to transcribe sung words from music recordings and is closely related to Automatic Speech Recognition (ASR). Although not specifically designed for lyrics transcription, the state-of-the-art ASR model Whisper has recently proven effective for ALT and various related tasks in music information retrieval (MIR). This paper investigates Whisper’s performance on Western classical music, using the “Schubert Winterreise Dataset.” In particular, we found that the average Word Error Rate (WER) with the unmodified Whisper model is 0.56 for this dataset, while the performance varies greatly across songs and versions. In contrast, spoken versions of the song lyrics, which we recorded, are transcribed with a WER of 0.14. Further systematic experiments with source separation and time-scale modification techniques indicate that Whisper’s accuracy in lyrics transcription is less affected by the musical accompaniment and more by the singing style.

## introduction lyrics, the words of a song, are vital to vocal music. they contain important information for listeners and bridge the gap between music and language. automatic lyrics transcription (alt) extracts these words, often from a mix of instruments and vocals @xcite . automatic speech recognition (asr) performs a similar task for normal speech @xcite . while both involve processing the human voice, speech, and singing differ in pitch fluctuations, pronunciation, speed, time variations, and vocabulary @xcite . musical accompaniment can further complicate alt, as it superimposes the singing voice, often with high temporal and spectral correlations @xcite . due to these differences, asr and alt have long been considered separate tasks @xcite . recent asr advances rely on large, diverse datasets and often use weakly-supervised or selfsupervised training @xcite . one state-of-the-art model, whisper, is trained on a total of 5 million hours of data @xcite . trained on such extensive data, whisper shows promising capabilities for alt as well. it can either be used without modifications @xcite , in combination with a large language model (llm) for transcript post-processing @xcite or be fine-tuned on specific music genres @xcite . understanding large pre-trained models is crucial, as these models can be useful for tasks with limited data like alt, in particular for underrepresented languages or genres @xcite . this paper aims to better understand whisper's alt performance and the challenges of transcribing singing compared to speech. different from the other works mentioned above, we focus on western classical music. in particular, we use the "schubert winterreise dataset" (swd) @xcite as a case study, which comprises nine complete recordings of the 24-song cycle "winterreise" by franz schubert. the winterreise is composed for solo voice with piano accompaniment, based on german poems from the early 19 th century. our contributions are twofold: an in-detail analysis of whisper's alt performance on the swd, and a comparison of speech and singing transcription through experiments with spoken versions of the lyrics, source separation, and time-scale modification. ## lyrics transcription results in this section, we evaluate the transcription performance of whisper for singing with accompaniment. figure 1 shows the wer of the whisper transcription of swd for each song and each version, along with the respective averages. the overall mean wer is µ = 0.56 but we can see considerable differences, both across songs and versions with an overall standard deviation of σ = 0.234. ## comparative analysis of speech and singing transcription in the previous section, we have seen that the alt performance of whisper for the classical music dataset swd is low compared to speech benchmarks. in this section, we further explore this difference, by investigating possible factors that may deteriorate the transcription performance from speech to classical singing. ## conclusion and future work our study investigates whisper's alt performance on western classical music using the swd as a case study. we find a higher wer for the swd compared to speech or other singing datasets, with significant fluctuations across songs and versions. vocabulary has a minor impact, as spoken lyrics wer is comparable to other speech benchmarks. mss-based vocal extraction has a negligible influence on the wer, indicating musical accompaniment is also not the primary issue. preliminary experiments show that whisper is robust against small speed variations but sensitive to larger variations in talking speed compared to normal speech. we hope our study serves as a starting point for analyzing how characteristics of speech and singing influence asr model performance. our evaluation methodology, though applied to whisper, is relevant beyond a single model. applying this approach to other asr models, such as @xcite , could enhance understanding of their behavior. this perspective positions our work as a case study for evaluating large audio models and highlights the potential of the music domain for thorough analysis of pre-trained models.