Adaptive Unsupervised Self-training for Disfluency Detection (2022)

Abstract: Supervised methods have achieved remarkable results in disfluency detection. However, in real-world scenarios, human-annotated data is difficult to obtain. Recent works try to handle disfluency detection with unsupervised self-training, which can exploit existing large-scale unlabeled data efficiently. However, their self-training-based methods suffer from the problems of selection bias and error accumulation. To tackle these problems, we propose an adaptive unsupervised self-training method for disfluency detection. Specifically, we re-weight the importance of each training example according to its grammatical feature and prediction confidence. Experiments on the Switchboard dataset show that our method improves 2.3 points over the current SOTA unsupervised method. Moreover, our method is competitive with the SOTA supervised method.

## introduction disfluency is a characteristic of spontaneous speech which is different from written texts. detecting and removing the non-fluent word sequences in spoken language transcripts can improve the transcripts' quality and provide clean inputs for the downstream nlp tasks, such as parsing, machine translation, and summarization @xcite . as shown in figure 1 , a standard annotation of the disfluency structure indicates the rm (reparandum, words that the speaker intends to discard), im (interregnum, filled pauses, discourse cue words, etc.), rp (repair, the associated repair) @xcite . most previous works @xcite @xcite @xcite on disfluency detection heavily relies on human-annotated corpora, which is difficult and expensive to obtain in practice. some researchers try to alleviate this issue with, for instance, self-supervised learning @xcite and semi-supervised learning techniques @xcite , but they still need a certain amount of human-annotated corpora to perform high performance. @xcite completely removes the need for human-annotated data. they first perform disfluency detection with the self-training method in an unsupervised manner, which shows promising performance. they leverage the pseudo data constructed by rules to train a weak disfluency detection model as a teacher. then, the student model is fine-tuned in an unsupervised manner, only using the pseudo-labels generated by the teacher. finally, the student model achieves promising performance. traditional self-training methods utilize the probability estimations to select high-confidence pseudo-labels for re-training, which encounters two problems. (1) selection bias. the strategy that selects samples with the high confidence pseudolabeled samples tends to neglect the hard and potentially informative samples. it results in the selftraining procedure converging to a suboptimal solution. (2) error accumulation. the strategy that selects pseudo-labels with probability estimations, limited by the ability of the model, will inevitably introduce a large number of noisy labels and the noise will be accumulated with the increase of iteration times. the above problems become the bottleneck for self-training, which prevents model performance from growing as the number of iter-ations increases. to alleviate the error accumulation problem, @xcite introduces a grammar checker into the selection process to enhance the ability to select high-quality pseudolabels, which can be seen as introducing external knowledge. concretely, @xcite employed a grammar checker to discard samples that may be noise based on their grammatical correctness. however, their method has no ability to recall potential informative samples, which leaves the selection bias problem unmitigated. in addition, due to the limited performance of the grammar checker, the error accumulation problem still exists to a certain extent. in this paper, we propose an adaptive selftraining method that utilizes a re-weighting strategy to address both the problem of selection bias and error accumulation. concretely, we assign specific weight to every pseudo-labeled sample based on its quality. we measure the quality of each sample by its grammatical feature and label confidence (see §2 for details). compared with the method that selects samples with high confidence, our reweighted method has two main advantages: first, instead of only selecting the pseudo-labeled samples with high confidence, our method makes all samples participate in the training process, which can alleviate the problem of selection bias to some extent. second, our method can reduce the negative effect of noise data by lowering the weights of lowquality pseudo-labels and emphasizing high-quality pseudo-labels during the self-training process. besides, to help improve the performance of self-training, we also propose a more powerful contrastive grammar checker. the grammar checker proposed by @xcite takes a single sentence as input and then judges whether the sentence is grammatical or not. in our practice, we found that the way of judging grammatical correctness by a single sentence alone faces great difficulties when using the asr results as input, e.g. there are incomplete sentences with missing beginnings or endings in the asr results, which may lead to the single-sentence grammar checker to misjudge. therefore, it is hard to train a high-performance single-sentence grammar checker model. to solve this problem, we employ a contrastive mechanism to make it easier to judge whether the sentence is grammatical or not. we take both the sentence before and after removing disfluency elements as the input of our grammar checker. therefore, our grammar checker can compare the two sentences to help judge whether the sentence is grammatical or not, which makes our grammar checker powerful. we evaluate the proposed method on the commonly used english switchboard dataset, and our method improves 2.3 points over the previous best result in unsupervised settings. moreover, our method achieves competitive performance compared to the state-of-the-art supervised method which utilizes 60k labeled sentences. in addition, the experimental result on three other datasets shows that our method can consistently achieve competitive performance compared to the supervised systems. the contributions of our work can be summarized as follows: we will release our code and model foot_0 . then, the teacher model is replaced with the student model. the process will be looped until the performance of the student model stops growing. ## ablation studies in this section, we analyze the effect of several components of our method. ## conclusion in this work, we propose an adaptive unsupervised method to deal with the task of disfluency detection in an unsupervised manner. we introduce a re-weighting mechanism into the self-training to alleviate the problems of selection bias and error accumulation. our experiments show that our reweighting mechanism can alleviate the problems of selection bias and error accumulation efficiently, and improve the performance of self-training for disfluency detection.