Investigating the Role and Impact of Disfluency on Summarization (2023)

Abstract: Contact centers handle both chat and voice calls for the same domain. As part of their workflow, it is a standard practice to summarize the conversations once they conclude. A significant distinction between chat and voice communication lies in the presence of disfluencies in voice calls, such as repetitions, restarts, and replacements. These disfluencies are generally considered noise for downstream natural language understanding (NLU) tasks. While a separate summarization model for voice calls can be trained in addition to chat specific model for the same domain, it requires manual annotations for both the channels and adds complexity arising due to maintaining two models. Therefore, it’s crucial to investigate if a model trained on fluent data can handle disfluent data effectively. While previous research explored impact of disfluency on question-answering and intent detection, its influence on summarization is inadequately studied. Our experiments reveal up to 6.99-point degradation in Rouge-L score, along with reduced fluency, consistency, and relevance when a fluent-trained model handles disfluent data. Replacement disfluencies have the highest negative impact. To mitigate this, we examine Fused-Fine Tuning by training the model with a combination of fluent and disfluent data, resulting in improved performance on both public and real-life datasets. Our work highlights the significance of incorporating disfluency in training summarization models and its advantages in an industrial setting.

## introduction disfluency is a prevalent feature of spontaneous spoken speech, encompassing natural interruptions during communication such as, repetitions (this is this is just not working), restarts (why don't you i will do it), and replacements (blue no no red). voice call interactions in contact center is rich in such disfluencies. thus, there is a growing need to understand the influence and impact of disfluencies on natural language processing tasks owing to an increase in such data containing disfluencies. existing research has primarily focused on disfluency detection @xcite and removal @xcite , with limited investigations into its impact on downstream tasks. recently, dao et al. (2022a) explored the effect of disfluency on intent and slot detection tasks, while @xcite finds that state-of-the-art pre-trained models are not robust when directly tested on the disfluent input for the task of question answering. however, the influence of disfluency on summarization models, particularly for spoken dialogues, remains relatively unexplored despite the extensive research on document and dialogue summarization. in this work, we investigate the impact of disfluencies on summarization. the contributions and key findings of our work are outlined below: why don't you actually i would suggest to look into this matter as this is a serious concern. repetition i don't want anyone using instant messaging. in this office. i don't don't want anyone using instant messaging. in this office. ## methodology in this section, we outline our method for curating a dataset of various types of disfluencies which is further used to analyze their impact on a summarization model trained on a fluent dataset. our investigation highlights a significant decrease in the performance of the model when presented with disfluent data. as a result, we discuss strategies to mitigate this impact and improve the model's ability to accurately summarize disfluent inputs. ## dataset to investigate the impact of disfluency in two different setups with varying characteristics, we utilize two publicly available datasets, dialogsum @xcite and debatesum @xcite . dialogsum is an abstractive summarization dataset which consists of a total of 13,460 dialogues demonstrating real-life scenarios across a variety of topics like schooling, work, medication, shopping, leisure, travel etc. with large scale data. the average number of tokens per dialog in dialogsum dataset is 131 while the compression ratio (ratio of the length of the summary divided by the length of the original text) is 0.18. on the other hand, debatesum is a word-level extractive summarization dataset containing a total of 187,386 unique pieces of evidences obtained from debates. for this dataset, average number of tokens in per sample of dataset is 372 while the compression ratio is 0.46. to evaluate the impact of the presence of disfluency, we generate the disfluency-induced versions of the dialogsum and debatesum corpora using lard as discussed in section 2.1. ## related works summarization is extensively studied with extractive @xcite and abstractive @xcite approaches being prominent, while some works utilize the best of both worlds in a hybrid approach @xcite . summarization has been explored in various document types, including shorttexts @xcite , dialogues @xcite , and medical conversations @xcite . while a number of works present summarization approaches tailored for conversations, to the best of our knowledge, the impact of disfluency on the summarization models is overlooked in the studies conducted so far. disfluency is widely studied in linguistics @xcite , psychology @xcite , and speech technology @xcite . works have focused on disfluency detection in speech @xcite and spoken transcripts @xcite . disfluencies have been shown to present challenges in nlp tasks, leading to the proposal of automatic disfluency removal systems @xcite . studies by @xcite and @xcite highlight the negative impact of disfluency on downstream tasks of intent detection, slot filling and question-answering tasks. our work falls in a similar category and is centered on investigating the influence of disfluency in summarization. to the best of our knowledge, this is the first study to examine how disfluencies and types of disfluencies affect various evaluation criteria for summarization. it is to be noted that our primary objective is not to propose a state-of-theart summarization model but to assess the impact of the presence of disfluency and investigate the effectiveness of simple mitigation techniques to alleviate its negative effects. we aspire to shed light on this overlooked area and encourage further research to develop robust summarization systems capable of handling all types of disfluencies. ## conclusion and future works in this work, we shed light on the adverse impact of the presence of disfluency on the performance of summarization models, for both abstractive and extractive summaries. as a mitigation strategy to lower the impact of disfluency, we investigate the approaches of cascaded pipeline and fused-fine tuning and observed that the cascaded pipeline which first removes the disfluent segments before passing it to the summarization model has a relatively inferior performance compared to finetuning a summarization model on a dataset containing both fluent and disfluent input. our investigation reveals that the mitigation strategy exhibits a relatively inferior performance on extractive summaries and on handling disfluency of the type 'replacement'. hence in future work we would like to carry out a more focused effort on mitigating the impact of disfluency for such cases.