Disfluencies in spontaneous speech are known to be associated with prosodic disruptions . However , most algorithms for disfluency detection use only word transcripts . Integrating prosodic cues has proved difficult because of the many sources of variability affecting the acoustic correlates . This paper introduces a new approach to extracting acoustic-prosodic cues using text-based distributional prediction of acoustic cues to derive vector z-score features ( innovations ) . We explore both early and late fusion techniques for integrating text and prosody , showing gains over a high-accuracy text-only model . Speech disfluencies are frequent events in spontaneous speech . The rate of disfluencies varies with the speaker and context ; one study observed disfluencies once in every 20 words , affecting up to one third of utterances ( Shriberg , 1994 ) . Disfluencies are important to account for , both because of the challenge that the disrupted grammatical flow poses for natural language processing of spoken transcripts and because of the information that they provide about the speaker . Most work on disfluency detection builds on the framework that annotates a disfluency in terms of a reparandum followed by an interruption point ( + ) , an optional interregnum ( { } ) , and then the repair , if any . A few simple examples are given below : Based on the similarity / differences between the reparandum and the repair , disfluencies are often categorized into three types : repetition ( the first example ) , rephrase ( the next example ) , and restart ( the last example ) . The interruption point is associated with a disruption in the realization of a prosodic phrase , which could involve cutting words off or elongation associated with hesitation , followed by a prosodic reset at the start of the repair . There may also be emphasis in the repair to highlight the correction . Researchers have been working on automatic disfluency detection for many years ( Lickley , 1994 ; Shriberg et al . , 1997 ; Charniak and Johnson , 2001 ; Johnson and Charniak , 2004 ; Lease et al . , 2006 ; Qian and Liu , 2013 ; Zayats et al . , 2016 ) , motivated in part by early work on parsing speech that assumed reliable detection of the interruption point ( Nakatani and Hirschberg , 1994 ; Shriberg and Stolcke , 1997 ; Liu et al . , 2006 ) . The first efforts to integrate prosody with word cues for disfluency detection ( Baron et al . , 2002 ; Snover et al . , 2004 ) found gains from using prosody , but word cues played the primary role . In subsequent work ( Qian and Liu , 2013 ; Honnibal and Johnson , 2014 ; Wang et al . , 2017 ) , more effective models of word transcripts have been the main source of performance gains . The success of recent neural network systems raises the question of what the role is for prosody in future work . In the next section , we hypothesize where prosody might help and look at the relative frequency of these cases and the performance of a high accuracy disfluency detection algorithm in these contexts . With the premise that there is a potential for prosody to benefit disfluency detection , we then propose a new approach to extracting prosodic features . A major challenge for all efforts to incorporate prosodic cues in spoken language understanding is the substantial variability in the acoustic correlates of prosody . For example , duration cues are expected to be useful -disfluencies are often associated with duration lengthening related to hesitation . However , duration varies with phonetic context , word function , prosodic phrase structure , speaking rate , etc . To account for some of this variability , various feature normalization techniques are used , but typically these account for only limited contexts , e.g. phonetic context for duration or speaker pitch range for fundamental frequency . In our work , we introduce a mechanism for normalization using the full sentence context . We train a sequential neural prediction model to estimate distributions of acoustic features for each word , given the word sequence of a sentence . Then , the actual observed acoustic feature is used to find the prediction error , normalized by the estimated variance . We refer to the resulting features as innovations , which can be thought of as a non-linear version of the innovations in a Kalman filter . The innovations will be large when the acoustic cues do not reflect the expected prosodic structure , such as during hesitations , disfluencies , and contrastive or emphatic stress . The idea is to provide prosodic cues that are less redundant with the textual cues . We assess the new prosodic features in experiments on disfluency detection using the Switchboard corpus , exploring both early and late fusion techniques to integrate innovations with text features . Our analysis shows that prosody does help with detecting some of the more difficult types of disfluencies . This paper has three main contributions . First , our analysis of a high performance disfluency detection algorithm confirms hypotheses about contexts where text-only models have high error rates . Second , we introduce a novel representation of prosodic cues , i.e. the innovation vector resulting from predicting prosodic cues given the whole sentence context . Analyses of the innovation distributions show expected patterns of prosodic cues at interruption points . Finally , we demonstrate improved disfluency detection performance on Switchboard by integrating prosody and textbased features in a neural network architecture , while comparing early and late fusion approaches . In this paper , we introduce a novel approach to extracting acoustic-prosodic cues with the goal of improving disfluency detection , but also with the intention of impacting spoken language processing more generally . Our initial analysis of a textonly disfluency detection system shows that despite high performance of such models , there exists a big gap in the performance of text-based approaches for some types of disfluencies , such as restarts and non-trivial or long rephrases . Thus , prosody cues , which can be indicative of interruption points , have a potential to contribute towards detection of more difficult types of disfluencies . Since the acoustic-prosodic cues carry information related to multiple phenomena , it can be difficult to isolate the cues that are relevant to specific events , such as interruption points . In this work , we introduce a novel approach where we extract relevant acoustic-prosodic information using textbased distributional prediction of acoustic cues to derive vector z-score features , or innovations . The innovations point to irregularities in prosody flow that are not predicted by the text , helping to better isolate signals relevant to disfluency detection that are not simply redundant with textual cues . We explore both early and late fusion approaches to combine innovations with text-based features . Our experiments show that innovation features are better predictors of disfluencies compared to the original acoustic cues . Our analysis of the errors and of the innovation features point to a limitation of the current work , which is in the modeling of F0 features . The current model obtains word-based F0 ( and energy ) features by simply averaging the values over the duration of the word , which loses any distinctions between rising and falling F0 . By leveraging polynomial contour models , we expect to improve both intonation and energy features , which we hope will reduce some of the false detections associated with emphasis and unexpected fluent phrase boundaries . An important next step is to test the system using ASR rather than hand transcripts . It is possible that errors in the transcripts could hurt the residual prediction , but if prosody is used to refine the recognition hypothesis , this could actually lead to improved recognition . Finally , we expect that the innovation model of prosody can benefit other NLP tasks , such as sarcasm and intent detection , as well as detecting paralinguist information .

Challenge: Text-based approaches for disfluency detection in speech have performance gaps for some disfluency types but integration of prosodic is challenging due to its high variability.
Approach: They propose a sequential neural model to estimate distributions of acoustic features for each word, given entire sentences and use them to find prediction errors.
Outcome: The proposed model with early and late fusions outperforms a text-only model on the Switchboard corpus however still limited in the modeling of F0 features.