In spoken dialogue systems, it is important for a system to know how likely a speech recognition hypothesis is to be correct, so it can reprompt for fresh input, or, in cases where many errors have occurred, change its interaction strategy or switch the caller to a human attendant. We have discovered prosodic features which more accurately predict when a recognition hypothesis contains a word error than the acoustic confidence score thresholds traditionally used in automatic speech recognition. We present analytic results indicating that there are significant prosodic differences between correctly and incorrectly recognized turns in the TOOT train information corpus. We then present machine learning results showing how the use of prosodic features to automatically predict correct versus incorrectly recognized turns improves over the use of acoustic confidence scores alone. One of the central tasks of the dialogue manager in most current spoken dialogue systems (SDSs) is error handling. The automatic speech recognition (ASR) component of such systems is prone to error, especially when the system has to operate in noisy conditions or when the domain of the system is large. Given that it is impossible to fully prevent ASR errors, it is important for a system to know how likely a speech recognition hypothesis is to be correct, so it can take appropriate action, since users have considerable difficulty correcting incorrect information that is presented by the system as true (Krahmer et al., 1999) . Such action may include verifying the user's input, reprompting for fresh input, or, in cases where many errors have occurred, changing the interaction strategy or switching the caller to a human attendant (Smith, 1998; Langkilde et al., 1999) . Traditionally, the decision to reject a recognition hypothesis is based on acoustic confidence score thresholds, which provide a reliability measure on the hypothesis and are set in the application (Zeljkovic, 1996) . However, this process often fails, as there is no simple one-to-one mapping between low confidence scores and incorrect recognitions, and the setting of a rejection threshold is a matter of trial and error (Bouwman et al., 1999) . Also, some incorrect recognitions do not necessarily lead to misunderstandings at a conceptual level (e.g. "a.m." recognized as "in the morning"). The current paper looks at prosody as one possible predictor of ASR performance. ASR performance is known to vary based upon speaking style (Weintraub et al., 1996) , speaker gender and age, native versus non-native speaker status, and, in general, the deviation of new speech from the training data. Some of this variation is linked to prosody, as prosodic differences have been found to characterize differences in speaking style (Blaauw, 1992 ) and idiosyncratic differences (Kraayeveld, 1997). Several other studies (Wade et al., 1992; Oviatt et al., 1996; Swerts and Ostendorf, 1997; Levow, 1998; Bell and Gustafson, 1999) report that hyperarticulated speech, characterized by careful enunciation, slowed speaking rate, and increase in pitch and loudness, often occurs when users in human-machine interactions try to correct system errors. Still others have shown that such speech also decreases recognition performance (Soltau and Waibel, 1998) . Prosodic features have also been shown to be effective in ranking recognition hypotheses, as a post-processing filter to score ASR hypotheses (Hirschberg, 1991; Veilleux, 1994; Hirose, 1997) . In this paper we present results of empirical studies testing the hypothesis that prosodic features provide an important clue to ASR performance. We first present results comparing prosodic analyses of correctly and incorrectly recognized speaker turns in TOOT, an experimental SDS for obtaining train information over the phone. We then describe machine learning experiments based on these results that explore the predictive power of prosodic features alone and in combination with other automatically available information, including ASR confidence scores and recognized string. Our results indicate that there are significant prosodic differences between correctly and incorrectly recognized utterances. These differences can in fact be used to pre- 218
REF demonstrate the additional use of prosodic features in deriving confidence scores, and transition the problem from one of word error rate to one involving concept error rate, which is more appropriate in the context of spoken dialogue systems.
Predicting Automatic Speech Recognition Performance Using Prosodic Cues
Applied Natural Language Processing Conference And Meeting Of The North American Association For Computational Linguistics
Computer Science