Disfluent Cues for Enhanced Speech Understanding in Large Language Models (2023)

Abstract: In computational linguistics, the common practice is to “clean” disfluent content from spontaneous speech. However, we hypothesize that these disfluencies might serve as more than mere noise, potentially acting as informative cues. We use a range of pre-trained models for a reading comprehension task involving disfluent queries, specifically featuring different types of speech repairs. The findings indicate that certain disfluencies can indeed improve model performance, particularly those stemming from context-based adjustments. However, large-scale language models struggle to handle repairs involving decision-making or the correction of lexical or syntactic errors, suggesting a crucial area for potential improvement. This paper thus highlights the importance of a nuanced approach to disfluencies, advocating for their potential utility in enhancing model performance rather than their removal.

## introduction when a speaker hesitates, interrupts themselves, repeats or corrects words, or abandons phrases, it can make their speech fragmented. listeners face a challenge called the "continuation problem" identified by @xcite . it means that listeners need to navigate through the linguistic obstacles to understand the speaker's true message. consider a situation where a listener encounters the statement, "i'd like a coffee with -uh, no, make that tea." to fully grasp this seemingly simple request, the listener needs to identify the disruption, understand its nature, and determine the intended message. this process is anchored in recognizing a three-part structure: the "reparandum" which is the portion of speech that's recognized as problematic or erroneous (in our example, "i'd like a coffee with"), the "edit interval" which is the break or hesitation that signals a correction is coming (in the scenario provided, the word "uh" serves this function), and the "repair interval" where the correction occurs (in the example, it's the "no, make that tea" segment) @xcite . when individuals correct themselves in speech, it's not a random occurrence. they notice a mismatch in their intended message, momentarily halt their speech, and then articulate a revised phrase. this self-correction considers the potential implications for the listener's understanding. the act of speech repair is a multifaceted process that includes aspects like message construction, formulation, articulation, parsing, and monitoring. various motivations can drive these self-repairs, such as rethinking the initial intent, reconsidering the mode of delivery, or rectifying inadvertent errors @xcite . the study of disfluencies has been prevalent in computational linguistics, largely aimed at enhancing recognition of spontaneous speech @xcite @xcite . traditionally, the approach has been to "clean" disfluent content for easier processing. yet, this work challenges the belief that removing disfluencies always aids comprehension. parallel to this, recent advances in natural language processing (nlp) have spotlighted promptbased models. these models, particularly when using "standard and clean" prompts, have shown a significant boost in zero-shot and few-shot performance compared to fine-tuned models @xcite @xcite . interestingly, these clean prompts offer semantically rich instructions, mirroring the rapid learning observed in humans when given clear instructions @xcite . our research involves a reading comprehension task with wikipedia articles, where we pose questions embedded with speech repairs and aim to pinpoint the answer within the text. we incorporated an additional set of tags to the dataset originally presented by @xcite to enrich our reading comprehension research. these tags were created specifically to distinguish between seven different categories of speech repairs. we applied each of these seven types of repairs to a subset of questions in the test set in addition to labeling the test set with these tags. we hypothesize that disfluent cues might be informative rather than just noise. for this study, we used a discrete prompt model inspired by @xcite . we ran evaluations on various pre-trained models, including bert, albert, t5 @xcite @xcite , and further incorporated gpt-3.5 and gpt-4. our findings challenge traditional perspectives on disfluencies, advocating a more nuanced approach where certain disfluencies can augment model performance. nevertheless, they also highlight the models' limitations in addressing repairs related to decision-making or lexical or syntactic corrections, underscoring a crucial area for potential enhancement. ## result our research has revealed several intriguing patterns related to the model's performance when trained with different types of repair-laden ques-tions. ## discussion large-scale language models (lms) are not robust when it comes to changing the mind midutterance ("do i want to say this now?") and making repairs for lexical or syntactic errors ("am i making an error?") our experiments have revealed that large-scale language models (lms) lack robustness when it comes to handling mind repairs, which involve the decision-making process of what to say at a given moment, and lexical/syntactic error repairs, which involve correcting potential errors in speech or writing. we found that when large-scale language models were directly tested on these types of disfluent input, their performance suffered a noticeable decline. the introduction of noise through the disfluent transformation proved to be a significant factor in diminishing the models' effectiveness. the dis-fluencies disrupted the flow and coherence of the input, challenging the models' ability to process and generate accurate responses. repairs resulting from context-based adjustments ("do i want to say it this way?") can benefit large-scale language models. repairs made as a result of adjustments based on contextual cues have the potential to benefit large-scale language models. our findings indicate that there is valuable information embedded within adjustment-based disfluencies that involves avoiding ambiguity, using appropriate level terminology, or maintaining coherence with previously used terms, which aids the models in their processing capabilities. notably, this advantage stemming from disfluencies becomes even more clear in a zero-shot setting, where the models exhibit enhanced performance when faced with unfamiliar or unseen data. these observations highlight the significance of leveraging adjustment-based disfluencies as a means to optimize the functionality and comprehension of large-scale language models. adjustments made for errors were conservative and aimed to stay faithful to the original expression, while repairs for appropriateness allowed more flexibility and required the creation of new concepts the investigation revealed notable variations in the methods used to make adjustments depending on the type of repairs required. in the case of actual errors, such as mistakenly mentioning "artur oppman" instead of "artur triton" or using "blue" instead of "red," the adjustments made were characterized by a highly conservative approach. the participants aimed to remain faithful to the original expression, resulting in relatively minor modifications to rectify the mistake. conversely, for appropriateness-related repairs, participants exhibited significant flexibility in their adjustments. this contrast in approach can be easily understood within the framework of the categories proposed in 3.1. when an error occurs, the same message, or a portion thereof, is essentially reprocessed. in contrast, repairing an inappropriateness often requires the creation of an entirely new concept or message, which consequently necessitates a fresh beginning. these disparities in the adjustment methods employed reflect the distinct nature of repairing actual errors versus addressing appropriateness issues. while errors involve solving specific inaccuracies ## conclusion we study the ability of large language models to handle disfluent inputs, revealing patterns and significant distinctions across different types of repairs-d-repairs, a-repairs, and e-repairs. our findings showed a significant decline in model performance when dealing with d-repairs compared to the fluent category. this trend was consistent across all models studied, including the advanced gpt-4, indicating an area with substantial potential for improvement. in contrast, arepairs often demonstrated performance on par with, or even exceeding, performance on fluent questions in some models, emphasizing the value of incorporating a-repairs into the training process. despite an overall decline in model performance when faced with e-repairs, we observed an exception within the phonological repairs subset, which displayed a positive trend as the shot values increased. these observations challenge conventional assumptions regarding disfluencies in speech data. contrary to the traditional view advocating for the elimination of disfluencies, our research supports a more nuanced perspective. we found that certain types of disfluencies, particularly adjustment-based repairs, can actually enhance model performance. however, our study also exposed a limitation in the robustness of large-scale language models when addressing repairs related to decision-making or the correction of lexical or syntactic errors. this find-ing underlines a crucial area where these models could be improved. identifying disfluencies parallels the challenge humans face when deciphering a disfluent utterance. typically, it's not until encountering the interregnum, or perhaps the onset of the repair, that we recognize the preceding content as marked "to be repaired," identifying it as the reparandum @xcite . in forthcoming research, we intend to look into the intricacies of incremental processing, aiming for real-time, left-to-right word analysis. however, it's crucial to underscore that our goal in this study wasn't to equate the model's processing with human real-time comprehension. our focus was to gauge the model's ability to handle disfluencies, akin to offline human processing.