Disfluency Generation for More Robust Dialogue Systems (2023)

Abstract: Disfluencies in user utterances can trigger a chain of errors impacting all the modules of a dialogue system: natural language understanding, dialogue state tracking, and response generation. In this work, we first analyze existing dialogue datasets commonly used in research and show that they only contain a marginal number of disfluent utterances. Due to this relative absence of disfluencies in their training data, dialogue systems may then critically fail when exposed to disfluent utterances. Following this observation, we propose to augment existing datasets with disfluent user utterances by paraphrasing fluent utterances into disfluent ones. Relying on a pre-trained language model, our few-shot disfluent paraphraser guided by a disfluency classifier can generate useful disfluent utterances for training better dialogue systems. We report on improvements for both dialogue state tracking and response generation when the dialogue systems are trained on datasets augmented with our disfluent utterances.

## introduction disfluencies are common interruptions in the flow of speech. in english, it is estimated that disfluencies account for 20% of the words @xcite and that there is a 50% probability that a sentence of 10-13 words will be disfluent @xcite . a probability that increases for longer sentences. since disfluencies are ubiquitous, they can have a significant impact on natural language processing (nlp) tasks. previous work has largely addressed disfluency detection and studied the impact of disfluencies in various nlp tasks @xcite . disfluency detection is a critical component of any nlp framework using speech transcriptions as input. disfluencies can mislead components of a dialogue system: natural language understanding (nlu), dialogue state tracking (dst), and response generation. on the other hand, disfluent utterances are usually absent in the publicly available dialogue datasets used for the research and development of dialogue systems. they are either removed, after disfluency detection or have never existed, for instance, in dialogue datasets made from non-spoken texts. the datasets on which dialog systems are trained and evaluated are often heavily curated. the dialogue systems trained on such datasets may then not be robust enough in real-world applications for which disfluent utterances are common. in this paper, we propose to augment existing training datasets with disfluent paraphrases to train a more robust dialogue system. in contrast to previous work on disfluency generation, our disfluent paraphraser only requires a very limited amount of training data that makes it applicable to a wide range of scenarios. our contributions are as follows: ## disfluency in dialogue disfluencies are usually categorized as in we can assume that depending on its category, a disfluency will not have the same impact on dialogue systems. for instance, "repair" and "restart" categories have more potential to mislead a system than "filled paused" since they may impact a large portion of an utterance. the example in to verify our assumption that most dialogue datasets used for research are not disfluent, we created a disfluency classifier (section 3.2) applied to publicly available dialogue datasets commonly used for training and evaluating dialogue systems. the classification results are presented in to train more robust dialogue systems, we augment their training data with synthetic disfluent utterances. while disfluency correction is a common task, there are only a few attempts in previous work for disfluency @xcite proposes to generate disfluency with a neural model inserting n-grams at specific positions in fluent sentences. they focus on two disfluency categories: "repair" and "repetition". their approach is able to generate natural disfluent sentences with a model trained on 29k disfluent sentences. in contrast, our approach relying on a paraphraser is able to generate any kind of disfluency but is not as conservative. our approach is not constrained to inserting tokens at specific positions. more recently, @xcite and @xcite proposed to generate disfluent sentences using heuristics. while their approaches are admittedly generating less natural disfluent sentences than with a neural model, they do not require to be trained and are able to generate disfluencies from any category covered by the heuristics. ## disfluency generation our disfluent paraphraser is applied to fluent utterances, identified by a disfluency classifier, from dialogue datasets. then, the disfluent utterances generated are added to the dialogue datasets and used to train more robust dialogue systems following a standard training pipeline. ## conclusion we demonstrated that our disfluent paraphraser generates useful disfluent paraphrases to better train dialogue models and especially improve their robustness to disfluent utterances. our approach improves dialogue state response and response generation for both fluent and disfluent user utterances. as future work, we would like to address the limitations discussed in section 6.