Improving Disfluency Detection by Self-Training a Self-Attentive Model (2020)

Abstract: Self-attentive neural syntactic parsers using contextualized word embeddings (e.g. ELMo or BERT) currently produce state-of-the-art results in joint parsing and disfluency detection in speech transcripts. Since the contextualized word embeddings are pre-trained on a large amount of unlabeled data, using additional unlabeled data to train a neural model might seem redundant. However, we show that self-training — a semi-supervised technique for incorporating unlabeled data — sets a new state-of-the-art for the self-attentive parser on disfluency detection, demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized word representations. We also show that ensembling self-trained parsers provides further gains for disfluency detection.

## introduction speech introduces challenges that do not appear in written text, such as the presence of disfluencies. disfluency refers to any interruptions in the normal flow of speech, including false starts, corrections, repetitions and filled pauses. @xcite defines three distinct parts of a speech disfluency, referred to as the reparandum, the interregnum and the repair. as illustrated in the example below, the reparandum the first kind of invasion of is the part of the utterance that is replaced or repaired, the interregnum uh i mean (which consists of a filled pause uh and a discourse marker i mean) is an optional part of the disfluency, and the repair the first type of privacy replaces the reparandum. the fluent version is obtained by removing the reparandum and the interregnum. @xcite the first kind of invasion of interregnum uh i mean the first type of privacy repair seemed invaded to me this paper will focus on joint disfluency detection and constituency parsing of transcribed speech. in the switchboard treebank corpus @xcite , which is a standard corpus for parsing studies on conversational speech, the reparanda, filled pauses and discourse markers are dominated by edited, intj and prn nodes, respectively (see figure 1 ). filled pauses and discourse markers belong to a finite set of words and phrases, so intj and prn nodes are trivial to detect @xcite . detecting edited nodes, however, is challenging and is the main focus of disfluency detection models. jamshid @xcite showed that a selfattentive constituency parser achieves state-of-theart results for joint parsing and disfluency detection. they observed that because the switchboard trees include both syntactic constituency nodes and edited nodes that indicate disfluency, training a parser to predict the switchboard trees can be regarded as multi-task learning (where the tasks are syntactic parsing and identifying disfluencies). in this paper, we extend the multi-task learning in jamshid @xcite to explore the impact of self-training @xcite and ensembling @xcite on the performance of the self-attentive parser. we aim to answer two questions about the state-of-the-art self-attentive parser: ## related work parsing speech transcripts is challenging for conventional syntactic parsers, mainly due to the presence of disfluencies. in disfluent sentences, the relation between reparandum and repair is different from other words in the sentence. the repair is usually a "rough copy" of the reparandum, using the same or similar words in roughly the same word order foot_0 @xcite . designed to capture tree-like structures, conventional syntactic parsers fail to detect "rough copies" which are strong indicators of disfluency. moreover, the reparandum and repair often do not form a syntactic phrase, which makes detecting the reparandum even harder. for these reasons, specialized disfluency detection models were developed to remove disfluencies prior to parsing (charniak and @xcite @xcite or special mechanisms were added to parsers to handle disfluencies @xcite @xcite . conventional parsing based models can use the syntactic location of the disfluency as a feature in a reranker @xcite . a similar gain can be achieved in neural models by training a joint parsing and disfluency detection model. in this multi-task learning setting, syntactic information helps the neural model detect disfluencies more accurately @xcite . state-of-the-art results for disfluency detection have been reported for transformer models using contextualized embeddings (e.g. elmo and bert) @xcite @xcite . the self-attention mechanism of the transformer is apparently effective for capturing "rough copy" dependencies between words. a recent study shows that prosody slightly improves the parsing performance of the selfattentive model over the text-only model, especially in long sentences @xcite . in this paper, we use a self-attentive model for joint disfluency detection and constituency parsing. disfluency detection models are usually trained and evaluated on the switchboard corpus. switchboard is the largest disfluency annotated dataset. however, only 5.9% of the words in the switchboard are disfluent @xcite . to mitigate the scarcity of labeled data, some studies have leveraged additional data by using: (i) contextualized embeddings pre-trained on enormous amount of unlabeled data @xcite @xcite and (ii) synthetic data generated by adding noise in the form of disfluencies to fluent sentences (e.g. repeating, deleting or inserting words in a sentence) @xcite @xcite . by contrast, this paper focuses on self-training, which is a simple semisupervised technique that has been effective in different nlp tasks, including parsing @xcite @xcite . to our best knowledge, this is the first work that investigates self-training a neural disfluency detection model. another technique commonly used for improving parsing is ensembling. ensembling is a model combination method, where scores of multiple models (they can be the same or different models, trained on the same or different data, with different random initializations) are combined in some way @xcite @xcite . the state-of-the-art for parsing written text is an ensemble of four bert-based self-attentive parsers, where the parsers are combined by averaging their span label scores @xcite . while ensembling is widely used in parsing, it has not been investigated for disfluency detection. in this paper, we also explore the impact of ensembling several parsing based disfluency detection models on disfluency detection performance. @xcite , we use a self-attentive constituency parser for joint disfluency detection and syntactic parsing 2 . the parsing model is based on the architecture introduced by @xcite , which is state-ofthe-art for (i) parsing written texts @xcite , (ii) parsing transcribed speech @xcite , and (iii) joint parsing and disfluency detection @xcite . ## experiments following @xcite , we split the switchboard into training, dev and test sets as follows: training data consists of the sw[23] * .mrg files, dev data consists of the sw4[5-9] * .mrg files and test data consists of the sw4[0-1] * .mrg files. all partial words 3 and punctuations are removed from the data, as they are not available in realistic asr applications @xcite . ## results we compare the performance of our best model with previous work on the switchboard test set. as demonstrated in we compare the performance of the selfattentive parser with state-of-the-art disfluency detection models. as shown in ## conclusion we introduced a new state-of-the-art for joint disfluency detection and constituency parsing of transcribed speech. we showed that self-training and ensembling are effective methods for improving disfluency detection. a qualitative analysis of the results also indicated that self-training is helpful for detecting complicated types of disfluencies, including corrections and restarts. in future work, we intend to explore the idea of self-training for parsing written texts. we also aim at integrating syntactic parsing and self-training more closely with automatic speech recognition. the first step is to develop parsing models that parse asr output, rather than speech transcripts.