Adversarial Training for Low-Resource Disfluency Correction (2023)

Abstract: Disfluencies commonly occur in conversational speech. Speech with disfluencies can result in noisy Automatic Speech Recognition (ASR) transcripts, which affects downstream tasks like machine translation. In this paper, we propose an adversarially-trained sequence-tagging model for Disfluency Correction (DC) that utilizes a small amount of labeled real disfluent data in conjunction with a large amount of unlabeled data. We show the benefit of our proposed technique, which crucially depends on synthetically generated disfluent data, by evaluating it for DC in three Indian languages- Bengali, Hindi, and Marathi (all from the Indo-Aryan family). Our technique also performs well in removing stuttering disfluencies in ASR transcripts introduced by speech impairments. We achieve an average 6.15 points improvement in F1-score over competitive baselines across all three languages mentioned. To the best of our knowledge, we are the first to utilize adversarial training for DC and use it to correct stuttering disfluencies in English, establishing a new benchmark for this task.

## introduction disfluencies are words that are part of spoken utterances but do not add meaning to the sentence. disfluency correction (dc) is an essential preprocessing step to clean disfluent sentences before passing the text through downstream tasks like machine translation @xcite . disfluencies can be introduced in utterances due to two main reasons: the conversational nature of speech and/or speech impairments such as stuttering. in real-life conversations, humans frequently deviate from their speech plan, which can introduce disfluencies in a sentence @xcite . stuttering speech consists of involuntary repetitions or prolongations of syllables which disturbs the fluency of speech. conversational disfluencies occur once every 17 words @xcite whereas a 2017 us study 1 shows that roughly 1% of the population stutters and predominantly consists of children. one out of every four children continues to suffer from this disorder lifelong. when such speech passes through an asr system, readability of the generated transcript deteriorates due to the presence of disfluencies in speech @xcite . @xcite defines the surface structure of disfluent utterances as a combination of reparandum, interregnum and repair. the reparandum consists of the words incorrectly uttered by the speaker that needs correction or complete removal. the interregnum acknowledges that the previous utterance may not be correct, while repair contains the words spoken to correct earlier errors. ## related work approaches in dc can be categorized into noisy channel-based, parsing-based, and sequence tagging-based approaches. noisy channel-based approaches rely on the following principle: a disfluent sentence y can be obtained from a fluent sentence x by adding some noise. these models try to predict the fluent sentence x given the disfluent sentence y @xcite @xcite . parsing-based approaches jointly predict the syntactic structure of the disfluent sentence along with its disfluent elements @xcite @xcite @xcite . sequence tagging-based approaches work on the following hypothesis: every word in a disfluent sentence can be marked as fluent/disfluent. these methods work best for shorter utterances and perform optimally for real-life conversational dc @xcite @xcite . moreover, sequence-tagging based methods require far less labeled data to perform well, compared to the other two methods. our approach to dc focuses on treating it as a sequence tagging problem rather than a machine translation task. the objective is to accurately classify each word as either disfluent or fluent, and create fluent sentences by retaining only the fluent words. the lack of labeled data for dc in low-resource languages has prompted the use of semi-supervised methods and self-supervised techniques @xcite . dc has also been studied as a component in speech translation systems, and thus its effect has been analyzed in improving the accuracies of machine translation models @xcite . synthetic data generation for dc has also received attention recently. these methods infuse disfluent elements in fluent sentences to create parallel data for training @xcite . our work is an extension of @xcite , which creates the first dataset for dc in bengali, hindi and marathi. we use this dataset to train our adversarial model to improve over the state-of-the-art in these languages. to the best of our knowledge, we are the first to model dc to correct stuttering asr transcripts. ## task 1: few shot dc in indian languages to test our proposed architecture, we train the model on the few-shot dc task for indian languages. the current state-of-the-art performance in bengali, hindi and marathi dc is obtained by training a large multilingual transformer model using synthetic data created by injecting disfluencies in fluent sentences using rules @xcite . we train our seq-gan-bert model using the authors' multilingual real and synthetic data. ## conclusion adversarial training using unlabeled data can benefit disfluency correction when we have limited amounts of labeled data. our proposed model can also be used to correct stuttering in asr transcripts with high accuracy. future work lies in integrating speech recognition models like whisper foot_3 or wav2vec 2.0 @xcite to create end-to-end speechdriven dc models. it will also be insightful to see how this model transfers to other low-resource languages with different linguistic properties. also like to thank nikhil saini for valuable discussions during the course of this project.