Harnessing Whisper for Prosodic Stress Analysis (2025)

Abstract: Prosody affects how people produce and understand language, yet studies of how it does so have been hindered by the lack of efficient tools for analyzing prosodic stress. We fine-tune OpenAI Whisper large-v2, a state-of-the-art speech recognition model, to recognize phrasal, lexical, and contrastive stress using a small, carefully annotated dataset. Our results show that Whisper can learn distinct, gender-specific stress patterns to achieve near-human and super-human accuracy in stress classification and transfer its learning from one type of stress to another, surpassing traditional machine learning models. Furthermore, we explore how acoustic context influences its performance and propose a novel black-box evaluation method for characterizing the decision boundaries used by Whisper for prosodic stress interpretation. These findings open new avenues for large-scale, automated prosody research. Models can be found at github.com/SSSohn/ProsodyBench.

## introduction prosody plays a crucial role in spoken language comprehension and production. it influences how listeners interpret words, sentences, and the pragmatic import of utterances, guiding syntactic disambiguation and affecting sentence processing efficiency. for example, prosodic cues can bias interpretations of syntactically ambiguous sentences and either strengthen or weaken garden paths, where listeners initially favor an incorrect interpretation before reanalyzing the sentence structure @xcite @xcite . beyond comprehension, prosody is also integral to speech production, as speakers unconsciously modulate their intonation, rhythm, and stress to convey different meanings @xcite . despite its importance, the study of prosody in both language processing and production remains relatively underdeveloped, largely due to the difficulty of analyzing prosodic features efficiently. traditional prosodic analysis relies on trained human annotators who manually label stress patterns in speech data (see @xcite )), which is a time-consuming and resource-intensive process that lacks scalability. this bottleneck limits large-scale investigations into prosodic variation and its interaction with lexical, syntactic, and discourse structures. a scalable and automated approach to prosodic analysis is therefore needed to advance our understanding of prosody and how it interfaces with other aspects of language. in this study, we explore the potential of openai's whisper large-v2 model @xcite , a state-of-the-art automatic speech recognition (asr) system, to recognize and analyze prosodic stress. although whisper was not originally trained for prosodic annotation, we demonstrate that fine-tuning it with a small, carefully curated dataset of stress-annotated utterances enables it to recognize different types of prosodic stress (i.e., phrasal, lexical, and contrastive stress) and transfer learned acoustic patterns between them. we further investigate the relationships between stress types based on how they facilitate or impede such transfer and how, for individual stress types, broader acoustic context can improve prosodic annotation to a superhuman level for both men and women. finally, we propose a novel black-box evaluation methodology for identifying acoustic decision boundaries that distinguish stress patterns, shedding light on how prosodic stress conveys meaning for men and women. ## preliminaries at its core, whisper leverages deep learning to analyze audio waveforms, extract patterns aligned with human speech, and decode these patterns into transcriptions @xcite . it is based on a transformer architecture @xcite trained through large-scale weak supervision to generalize across diverse acoustic environments, speakers, and linguistic contexts. ## related work as a baseline for model comparison, we use the knutsen and stromswold study @xcite , from which the fine-tuning dataset was derived. they examined gender differences in the acoustic realization of phrasal, lexical, and contrastive stress, addressing a gap in prior research on prosodic variation between men and women. acoustic features (including pitch, amplitude, and duration) were extracted and analyzed using bayesian anovas, random forest classification (rfc), and bayesian mixed-effects regression to determine their relative importance in signaling stress. their results indicate that while both men and women employ pitch (measured by fundamental frequency f0), amplitude, and duration to mark stress, their reliance on these features differs systematically. ## extent of acoustic context the rfc baseline is limited in that the sentenceembedded minimal pairs for phrasal and contrastive stress do not leverage acoustic features outside the minimal pair, which is the region of interest (roi) bracketed in we show that the first syllable/morpheme (e.g., "in-", "green-", and "black") and the second syllable/morpheme (e.g., "-sult", "-house", and "cow") hold different amounts of prosodic information. for both men and women, the second syllable/morpheme holds significantly more information than the first ( ## transfer between stress types in addition to handling varying acoustic contexts with ease, whisper is able to generalize its learnings across diverse acoustic environments, speakers, and linguistic contexts. given its superior performance to rfc models ( to this end, we first fine-tune a control model using all types of stress from a single random control participant. this equips whisper with the minimum knowledge needed to learn the unique lexicons in our fine-tuning dataset (i.e., the capitalization of stressed syllables). for consistency between stress types, we only use the rois ( ## characterizing decision boundaries unlike rfc models, which provide interpretable feature importance scores, whisper's learned features are more challenging to extract. to this end, we propose a black-box evaluation methodology, which can be applied to any stress type and any prosodic annotation model. for each stress type, the minimal pairs can be separated into two categories based on their canonical stress patterns: stress-first or stress-final ( we use the stress-first category for each stress type and perturb its pitch (p1), amplitude (p2), and pause duration (p1 and p2). in order to ground these perturbations, we first consider the distributions of pitch and amplitude shifts that participants produce for the first and second syllables/morphemes of their minimal pairs as well as pause duration shifts across both syllables/morphemes ( for p2 (figure 2 ), we observe a linear decision boundary along the axis of amplitude divergence (bottom-left to top-right) for lexical and contrastive stress, where stress-first accuracy decreases as the first amplitude decreases and second amplitude increases. the decision boundary is strongest for lexical stress and moderately strong for contrastive stress with low sensitivity to pause duration. similar to pitch shift, the linear decision boundary is only apparent for phrasal stress when pause duration increases by 0.06, which reduces its credibility. however, its high sensitivity to pause duration still holds true. ## discussion the successful application of whisper to prosodic stress analysis enables large-scale studies of spoken language processing and production that account for prosody's role in communication. ## conclusion whisper demonstrates near-human and superhuman capabilities for recognizing prosodic stress, harnesses variable-length acoustic context with ease, and transfers learned acoustic patterns be-tween broader stress types, greatly surpassing prior work in accuracy and robustness. the final gap between whisper and prior work was in interpretability, which we have addressed with our black-box evaluation methodology. this method elucidates the nuanced interplay of acoustic features, which importance scores only convey coarsely, and it makes no assumptions about the model, meaning that all models can be evaluated in a standardized manner. with proper fine-tuning using a very small, carefully curated dataset, whisper could become a promising tool for cross-linguistic prosodic research, potentially illuminating questions about cross-language and language-specific patterns in stress. the fine-tuned whisper model weights (trained on the full dataset and the less noisy subset for all stress types) are available at github.com/sssohn/prosodybench.