DISCO : A Large Scale Human Annotated Corpus for Disfluency Correction in I ndo- E uropean Languages (2023)

Abstract: Disfluency correction (DC) is the process of removing disfluent elements like fillers, repetitions and corrections from spoken utterances to create readable and interpretable text. DC is a vital post-processing step applied to Automatic Speech Recognition (ASR) outputs, before subsequent processing by downstream language understanding tasks. Existing DC research has primarily focused on English due to the unavailability of large-scale open-source datasets. Towards the goal of multilingual disfluency correction, we present a high-quality human-annotated DC corpus covering four important Indo-European languages: English, Hindi, German and French. We provide extensive analysis of results of state-of-the-art DC models across all four languages obtaining F1 scores of 97.55 (English), 94.29 (Hindi), 95.89 (German) and 92.97 (French). To demonstrate the benefits of DC on downstream tasks, we show that DC leads to 5.65 points increase in BLEU scores on average when used in conjunction with a state-of-the-art Machine Translation (MT) system. We release code to run our experiments along with our annotated dataset here.

## introduction humans often think and speak simultaneously in conversations, introducing erroneous words in utterances @xcite . these words do not contribute to semantics of a sentence and hence can be removed to create fluent and easy-to-interpret utterances. disfluency correction (dc) is defined as the removal of such disfluent elements from spoken utterances @xcite . motivation: apart from making sentences readable and interpretable, dc also helps downstream natural language processing tasks like machine translation (mt) @xcite . removing disfluencies shortens sentences, making it easier for automatic mt systems to translate these utterances. moreover, the removed erroneous words are not translated which makes the output translation fluent containing all semantics from the source sentence. ## related work the study of disfluencies as a spoken language phenomenon was first proposed in @xcite . dc has been established as a vital post-processing task for asr transcripts @xcite . although earlier dc systems were based on translation methods @xcite , current research covers two additional methods: parsing-based and sequence taggingbased techniques. translation-based methods use a noisy channel approach towards dc hypothesizing that disfluent sentences are fluent sentences with noisy elements @xcite . parsing-based methods use techniques such as dependency parsing to predict syntactic structure of an utterance along with disfluent elements (rasooli and tetreault, 2015; honnibal and @xcite . sequence tagging methods work well for disfluency removal from real-life spoken utterances, assigning disfluent/fluent label to every word in the sentence @xcite . language clues and part-of-speech tags based systems have also been explored for dc @xcite . there is a notable gap in literature regarding real data annotation in dc, with switchboard @xcite and @xcite being the most extensive open-source labeled datasets for english dc. although @xcite introduced a dataset for disfluencies in english question answering, they have not been annotated for disfluent words. without labeled data, various zero-shot, few-shot, and multi-task learning techniques have been proposed, which train on multilingual data, creating and utilizing synthetically generated disfluent sentences (wang et al., @xcite . in this work, we experiment with sequence tagging methods for dc. ## disco: a dataset for disfluency correction this section analyzes the disco corpus, created with the help of english, hindi, german and french language experts. disco contains paral-lel disfluent-fluent sentence pairs in the above four languages and english translations of fluent sentences in hindi and german along with disfluency and domain labels. @xcite defines disfluencies as a composition of reparandum, interregnum and repair (figure 1 ). reparandum refers to words erroneously uttered by the speaker. the speaker acknowledges that a previous utterance might be incorrect using interregnum, whereas repair contains words that correct mis-spoken words. disfluent utterances might consist of an interruption point-a spoken phenomena like speech pauses. dc removes reparandum and interregnum while retaining repair to make the output sentence more fluent. ## dataset evaluation this section describes the experiments we perform to evaluate the disco corpus. our evaluation strategy measures the efficacy of the corpus for robust disfluency correction in a wide variety of cases. moreover, we also test the ability of our trained models to correct disfluencies for improving downstream machine translation. ## results and analysis we thoroughly analyse all experiments performed in dc. this section also discusses some case studies highlighting strengths and weaknesses of our best models. our experiments in analyzing the impact of dc on mt provides interesting linguistic insights into the phenomenon of disfluencies. ## conclusion and future work this paper introduces the disco dataset for disfluency correction in four widely spoken languages: english, hindi, german and french. our work highlights the importance of large-scale projects in nlp that scale the amount of labeled data available. spoken interactions between humans and ai agents are riddled with disfluencies. eliminating disfluencies not only improves readability of utterances but also leads to better downstream translations. our dataset, which consists of roughly 3000 parallel disfluent-fluent sentences in each language, significantly reduces the data scarcity problem in dc. this allows training of large transformer models to correct spoken disfluencies from written transcripts with high accuracy. lack of conversational translation datasets has led to most mt systems trained on fluent text. our experiments show that such models if used in conversational settings do not perform well. by adding a dc model in the pipeline, which is often a smaller model with an incremental increase in latency, one can improve the downstream translations outputted by an mt system that does not adjust to conversational phenomena. moreover, our dataset in german -english and hindi -english can also be used to finetune conversational mt models. future work lies in experimenting with better ml models for sequence tagging-based dc supporting multilingual training. these should also incorporate linguistic features like reparandum, interregnum and repair. multimodal dc presents a promising direction as it has the capability of using both speech and text features for correction tasks @xcite . additionally, trained dc models must be evaluated using diverse samples from various domains and dialects. special efforts must be made to collect disfluent speech transcripts to be annotated and trained for dc in other low-resource languages.