Combining Self-Training and Self-Supervised Learning for Unsupervised Disfluency Detection (2020)

Abstract: Most existing approaches to disfluency detection heavily rely on human-annotated corpora, which is expensive to obtain in practice. There have been several proposals to alleviate this issue with, for instance, self-supervised learning techniques, but they still require human-annotated corpora. In this work, we explore the unsupervised learning paradigm which can potentially work with unlabeled text corpora that are cheaper and easier to obtain. Our model builds upon the recent work on Noisy Student Training, a semi-supervised learning approach that extends the idea of self-training. Experimental results on the commonly used English Switchboard test set show that our approach achieves competitive performance compared to the previous state-of-the-art supervised systems using contextualized word embeddings (e.g. BERT and ELECTRA).

## introduction automatic speech recognition (asr) outputs often contain various disfluencies, which is a characteristic of spontaneous speech and create barriers to subsequent text processing tasks like parsing, machine translation, and summarization. disfluency detection @xcite @xcite focuses on recognizing the disfluencies from asr outputs. as shown in figure 1 , a standard annotation of the disfluency structure indicates the reparandum (words that the speaker intends to discard), the interruption point (denoted as '+', marking the end of the reparandum), an optional interregnum (filled pauses, discourse cue words, etc.) and the associated repair @xcite . ignoring the interregnum, disfluencies are categorized into three types: restarts, repetitions and corrections. our model builds upon the recent work on noisy student training @xcite , a semisupervised learning approach based on the idea of experimental results on the commonly used english switchboard set show that our approach achieves competitive performance compared to the previous state-of-the-art supervised systems using contextualized word embeddings (e.g. bert and electra). besides the experiment on the commonly used english switchboard set, we evaluate our approach on another three different speech genres, and also achieve competitive performance compared to the supervised systems using contextualized word embeddings. ## ablation studies in this section, we study the importance of grammaticality judgment model and iterative training. ## conclusion in this work, we explore unsupervised disfluency detection by combining self-training and selfsupervised learning. we showed that it is possible to completely remove the need of human-annotated data and train a high-performance disfluency detection system in a completely unsupervised manner. 61772153. wanxiang che is the corresponding author.