PSST ! Prosodic Speech Segmentation with Transformers (2023)

Abstract: We develop and probe a model for detecting the boundaries of prosodic chunks in untranscribed conversational English speech. The model is obtained by fine-tuning a Transformer-based speech-to-text (STT) model to integrate the identification of Intonation Unit (IU) boundaries with the STT task. The model shows robust performance, both on held-out data and on out-of-distribution data representing different dialects and transcription protocols. By evaluating the model on degraded speech data, and comparing it with alternatives, we establish that it relies heavily on lexico-syntactic information inferred from audio, and not solely on acoustic information typically understood to cue prosodic structure. We release our model as both a transcription tool and a baseline for further improvements in prosodic segmentation.

## introduction a growing body of research in phonetics, phonology, and speech processing focuses on prosody: the encoding of prominence and phrasal organization @xcite through interconnected suprasegmental cues (intonation, stress, rhythm, etc.) @xcite . one reason for this focus is that prosodic phrasing groups words into chunks that can facilitate the generation and processing of naturalistic running speech for both speakers and listeners. for example, in english, the presence of detectable boundaries between chunks enhances speech intelligibility @xcite and helps listeners correctly discern the syntactic structure of the utterance @xcite @xcite @xcite . in this paper, we generate, evaluate, and probe machine-learned models for detecting the boundaries of prosodic chunks in untranscribed conversational english speech. we focus on boundaries of 1 https://github.com/nathan-roll1/psst the intonation unit (iu), which delineate "chunks" of speech that reflect cognitive and prosodic cohesion @xcite . developing a robust boundary detector for conversational speech would have important implications for linguistics. methodologically, it would open the door to automated systems for fine-grained discourse transcription, and theoretically, it would facilitate exploration of the way that suprasegmentals interact to cue prosodic structure @xcite . given the utility of prosodic boundaries for human speech perception, it may also contribute to the robustness of automatic speech recognition (asr) generally for conversational speech. robust conversational asr is made difficult by the fact that cues to segmental information are often reduced in conversation, may be masked by significant interspeaker variation, and often do not correspond precisely to the rigid syntactic structures of written language, among other challenges. the detection of prosodic boundaries via automated methods has a rich history in work that aims to segment transcriptions of speech. however prior works have largely taken a pipeline approach: first creating textual transcriptions (either manually or via asr) and subsequently applying boundary detection methods to the generated transcript. in addition, they have not typically focused on identifying iu boundaries in everyday conversations. many works (e.g., @xcite @xcite use the switchboard corpus to identify syntactically-based prosodic boundaries in telephone conversations between strangers, using orthographic inputs and/or manually crafted acoustic features. @xcite applies pause, pitch, energy, and duration information to a similar task in spoken mandarin. more recent work has pursued integrated approaches that consider speech-to-text (stt) transcription and segmentation simultaneously, but still have not focused on iu boundaries in conversational speech. @xcite introduced a model to perform asr, segmentation, and diarization concurrently on the librispeech corpus of read speech. similarly, @xcite detected phone-and wordlevel timestamps while performing asr on the timit and wsj corpora of read speech. here, we follow this more recent work in taking an integrated approach, which we use to detect iu boundaries in everyday conversations. we develop an end-to-end model that incorporates iu boundary detection into a transformer-based @xcite stt task. specifically, we fine-tune whisper @xcite , a highly successful stt model, to generate iu boundaries as it processes audio and generates a transcription. the incorporation of iu boundary detection into stt transcription allows for counterfactual considerations of lexico-syntactic probabilities, and allows our model to recognize the strong correspondences and interactions between syntax and prosody that are fundamental to linguistic theory @xcite . studies on automatic boundary predictions in the prosodic domain have primarily concentrated on two key areas: feature engineering and modeling methods. feature engineering (e.g., ananthakrishnan and narayanan, 2005) involves identifying and operationalizing acoustic features such as pitch as pause that correlate with prosodic boundaries. modeling methods involves comparing various statistical machine learning frameworks -such as memory-based learning @xcite , maximum entropy @xcite , and deep neural networks @xcite -that use these features in different ways to identify prosodic boundaries in unlabeled data. the transformer architecture obviates the distinction between these areas by allowing the model to discover useful acoustic features itself, based on self-attention mechanisms applied to positionallyencoded audio data. the model therefore efficiently discovers and leverages rich features present in input audio, without enforcing strong assumptions about what those features are or how they are structured in the time or frequency domains. in doing so, it exhibits similarity to human iu boundary detection by considering a myriad of fine-grained cues, including those that are difficult to operationalize with direct feature engineering @xcite . this represents a significant departure from previous attempts to detect prosodic phrase boundaries, which have typically used either simple durational cues and pauses @xcite or a combination of other predetermined suprasegmental cues @xcite , and/or have isolated the task of prosodic boundary detection from that of stt transcription @xcite . we investigate whether fine-tuning on a small, high quality dataset can "teach" a pretrained transformer-based stt model to segment conversational speech audio into ius, by detecting iu boundaries in the course of transcription. we perform two experiments, with the following research objectives: ## experiment 1: reliable iu detection in experiment 1, we fine-tune whisper @xcite , a transformer-based stt model, to identify iu boundaries as it processes and transcribes audio. our goal is not to improve the basic word recognition rate of whisper, but rather to investigate whether its capabilities can be leveraged to recognize intonation unit boundaries, in a generalizable way. the model is fine-tuned on a corpus of conversational american english, and we establish its performance on held-out data from the same corpus. then, we assess its robustness to naturalistic acoustic variation and differences in prosodic transcription protocol, by evaluating it on out-of-distribution speech data (i.e., non-american english data not used in the training of the model) from a corpus of conversational british english that uses distinct criteria to determine iu boundaries. ## experiment 2: understanding the model in experiment 2, we explore factors that contribute to psst's excellent results. in experiment 2a, we explore the kind of acoustic features that the model may be relying upon, by evaluating performance on acoustically degraded stimuli. in experiment 2b, we explore the extent to which the model integrates acoustic and lexico-syntactic information, by comparing its performance with that of alternatives that have limited integration. ## discussion & conclusion this study had two research objectives, as stated in section 1. in relation to objective 1, we successfully fine-tuned whisper @xcite to segment conversational speech into ius. we achieved f-scores of 0.87 on held-out test data and 0.72 on out-of-distribution data, indicating strong reliability. whisper was originally trained on the simple objective of discerning words from audio, yet the fact that we were able to repurpose it successfully using few-shot learning holds significant promise for other nlp studies that rely on smaller datasets. in relation to objective 2, we explored the potential factors influencing the model's performance. our findings suggest that the model uses acoustic information primarily for lexical identification. interestingly, the model also appears to benefit from the interactions between acoustic and lexico-syntactic information that are made possible through the integration of iu boundary detection with stt transcription. these results may be surprising from an expectation that prosodic boundaries would be reflected primarily by acoustic cues, but they reinforce the understanding from linguistic theory that prosody involves complex interactions between syntax and phonology @xcite . given these results, there are two clear next steps. first, though our model was able to perform reliable iu boundary detection, its performance was hindered in out-of-distribution contexts involving different dialects and transcription protocols. expanding the training set to be more representative of such variation would further improve its reliability and adaptability. second, though we observed a benefit from integrating acoustic and lexico-syntactic information, it appears that the acoustic information was relatively underweighted. this is likely a reflection of the fact that finetuning the integrated model represents a very small amount of training relative to training the original stt model, in which acoustic cues to prosodic boundaries have limited relevance. fine-tuning for longer, or on more data, may help increase the weight of acoustic cues to prosodic boundaries. in addition, experiments with acoustically enhanced rather than degraded stimuli may help to illuminate the circumstances under which acoustic cues to prosodic boundaries can override biases from lexico-syntactic information. our results suggest that stt transcription and prosodic boundary identification should not be approached as independent challenges, but rather as interacting components of a unified speech processing objective. simply requiring prosodic features to be represented in the desired output transcriptions unlocks a seemingly latent ability for stt models to identify them. overall, our results suggest that such stt models implicitly represent prosodicallyrelevant information, given their success in a fewshot context. furthermore, the robustness of segmentation performance when exposed to moderate frequency-based signal tampering, or even complete f0 masking, strengthens the case for prosodysyntax interplay at the "heart" of high-performance asr models. by following a similar process to what we have shown here, there is strong potential for stt models to be extended to detect other speech phenomena as well -such as prosodic accents, vocal quality changes, or even environmental contexts -which would put us one step closer to a fully automated discourse transcription system.