End-to-End Speech Recognition and Disfluency Removal (2020)

Abstract: Disfluency detection is usually an intermediate step between an automatic speech recognition (ASR) system and a downstream task. By contrast, this paper aims to investigate the task of end-to-end speech recognition and disfluency removal. We specifically explore whether it is possible to train an ASR model to directly map disfluent speech into fluent transcripts, without relying on a separate disfluency detection model. We show that end-to-end models do learn to directly generate fluent transcripts; however, their performance is slightly worse than a baseline pipeline approach consisting of an ASR system and a specialized disfluency detection model. We also propose two new metrics for evaluating integrated ASR and disfluency removal models. The findings of this paper can serve as a benchmark for further research on the task of end-to-end speech recognition and disfluency removal in the future.

## introduction disfluency is a characteristic of spontaneous speech which is not present in written texts. disfluencies include filled pauses (e.g. um and uh), repetitions (e.g. the the), corrections (e.g. show me the flights . . . the early flights), parenthetical asides (e.g. you know), interjections (e.g. well and like), restarts (e.g. there's a . . . let's go) and partial words (e.g. wouand oper-) which frequently occur in spontaneous speech foot_0 and reduce the readability of speech transcripts @xcite . they also pose a major challenge to downstream tasks relying on the output of speech recognition systems, such as parsing and machine translation models @xcite @xcite . since these models are usually trained on fluent clean corpora, the mismatch between the training data and the actual use case decreases their performance. to tackle this challenge of spontaneous speech, specialized disfluency detection models are developed and applied as a post-processing step to remove disfluencies from the output of speech recognition systems @xcite @xcite . one type of disfluency which is especially problematic for disfluency detection models is speech repair. @xcite defines three distinct parts of a speech repair, referred to as reparandum, interregnum and repair. as illustrated in the example below, the reparandum to boston is the part of the utterance that is replaced and is usually followed by an interruption point in the speech signal, the interregnum uh i mean is an optional part of a disfluent structure (that consists of a filled pause uh and a discourse marker i mean) and the repair to denver replaces the reparandum. the fluent version is obtained by deleting reparandum and interregnum words. i want a flight to boston reparandum uh i mean interregnum to denver. repair disfluency detection is usually an intermediate step between an asr model and a downstream task. this pipeline approach is complex to implement and leads to higher inference latency. it also has the potential problem of errors compounding between components, e.g. recognition errors lead to larger disfluency detection errors. end-to-end models, on the other hand, are less prone to such problems. more importantly, end-to-end models can leverage paralinguistic features in speech signal that are not available in pipeline systems. speech carries extra information beyond the words which might provide useful cues to disfluency detection 2 . in this paper, we address the task of end-to-end speech recognition and disfluency removal. specifically, we investigate whether it is possible to train an asr model end-to-end to directly map disfluent speech into fluent transcripts, without an intermediate disfluency detection step. some previous work has attempted disfluency detection as part of another task in an end-to-end manner, e.g. joint disfluency detection and constituency parsing @xcite and direct translation from disfluent spanish speech to fluent english transcripts @xcite . however, to the best of our knowledge, this is the first work that systematically investigates the task of end-to-end asr and disfluency removal, serving as a starting point for future research into end-toend disfluency removal systems. in this paper, we aim to answer the following questions: ## related work disfluency removal is typically performed by training a specialized disfluency detection model on disfluency labeled data and applying it as a separate component following an asr model and prior to a downstream task. the specialized disfluency detectors @xcite @xcite are usually trained on the switchboard corpus @xcite which is the largest available dataset with gold (i.e. human-annotated) disfluency labels. state-ofthe-art disfluency detectors use transformer models with pretrained contextualised word embeddings (e.g. bert) @xcite @xcite @xcite . multi-task learning has been effective for disfluency detection, for example, a transformer trained to jointly detect disfluencies and find constituency parse trees would leverage syntactic information and detect disfluencies more accurately @xcite . self-training and ensembling have also shown to provide benefit to disfluency detection @xcite . selftraining on disfluent data provides benefits orthogonal to the pretrained contextualized embeddings and mitigates the scarcity of gold disfluency labeled data. the bert-based self-attentive parser introduced in jamshid @xcite is the current state-of-the-art in disfluency detection; thus, we use it as the "off-the-shelf" disfluency detector in our pipeline approach, as explained in section 5. with the rise of end-to-end models, the conversational speech translation models that directly translate disfluent speech into fluent texts have recently attracted increasing attention @xcite @xcite . the most similar previous work to ours is @xcite . they train a sequence-tosequence model (called fluent model) to directly translate from disfluent spanish speech to fluent english transcripts without a separate disfluency detection step. as a baseline, they train a model (called disfluent model) on disfluent speech and disfluent translations. to compare the performance of the fluent and disfluent models, they score the outputs against the fluent references using bleu and meteor. similar meteor scores are reported for both models, but bleu scores are lower with the disfluent model. they argue that the disfluencies generated by the disfluent model lead to n-gram break-up in the fluent references and consequently decrease the bleu scores. they conclude that higher bleu scores in the fluent model imply that it is better at generating fluent translations. although bleu and meteor are standard metrics for evaluating machine translation systems, they are not designed to evaluate the performance of end-to-end models in terms of disfluency removal. since many disfluent words are copies of fluent words, the bleu score of disfluent transcripts (e.g. the the snack was delicious) can be higher than that of fluent transcripts containing translation errors (e.g. the meal was delicious against the fluent reference the snack was delicious). furthermore, these metrics are sensitive to sequence length which makes them undesirable for evaluating end-to-end models incorporating disfluency removal. fluent transcripts tend to contain fewer tokens per sentence in comparison with disfluent transcripts. by contrast, we introduce two new metrics in this paper that systematically measure the fluency of the generated transcripts. we also benchmark our end-to-end model against a state-of-the-art pipeline approach to explicitly evaluate its disfluency detection performance. ## evaluating integrated asr and disfluency models the performance of asr models is usually evaluated in terms of word error rate (wer). wer is calculated by finding an alignment between the reference transcript (which is human-transcribed speech) and asr output so that a minimum number of edits (i.e. substitutions, insertions and deletions) are required for transcribing the asr output to the reference transcript. given an alignment, wer is the ratio between the number of incorrectly aligned words and the total number of words in the reference transcript: to address the limitations of the existing metric, we introduce two new evaluation metrics foot_7 which assess the output of an integrated model in terms of fluency and word recognition accuracy in section 4.1. we then demonstrate the problems associated with the standard asr alignment algorithm and how it can lead to undesirable alignments for evaluating integrated asr and disfluency models. as a solution, we modify standard alignment weights to correctly align reference transcripts (which may contain disfluencies) with integrated model outputs in section 4.2. ## experiments we train our asr models on two corpora of english conversational telephone speech: (i) switchboard-1 release 2 (swbd) @xcite ) and (ii) fisher part 1 @xcite and part 2 @xcite . switchboard-1 release 2 is a collection of about 2,400 telephone conversations (260 hours of speech), of which 1,126 conversations were hand-annotated with disfluencies as part of the penn treebank release 3 dataset @xcite , which we refer to as gold data. the original release of switchboard does not contain time-alignment annotations which are required for preparing the asr training data. mississippi state university researchers ran a clean-up project on switchboard-1 release 2 and produced accurate time alignments foot_9 which we use for speech segmentation. fisher part 1 and 2 are a collection of 11,700 telephone conversations (total 2,000 hours of speech), which contain time-aligned transcripts, but no disfluency annotations. to identify the disfluencies in the fisher data and the portion of the swbd data with no gold disfluency labels, we use an "off-the-shelf" state-of-the-art disfluency detection model 10 (jamshid lou et al., 2019). we call the automatically annotated data silver data. the disfluency detection model used to obtain silver data is a bert-based self-attentive parser that jointly ## results we compare the performance of our integrated asr and disfluency models (trained on fluent transcripts) to the baseline pipeline models consisting of the asr models (trained on disfluent transcripts) combined with the "off-the-shelf" disfluency detection model. as shown in fer measures the (fluent) word recognition performance of the model while der reflects how well the model performs in terms of disfluency detection and removal. the baseline asr models (without disfluency detection) have the lowest error rate on fluent areas (i.e. fer). however, when we apply the "off-the-shelf" disfluency detection model on the output of the baseline asr models, fer significantly increases, indicating that errors made by the "off-the-self" disfluency detection model harm the detection of fluent words. the fluent error rate of the end-to-end models is lower than the pipeline models. comparing the disfluent error rate of the end-to-end and baseline models, we realize that simply training an asr model on disfluent speech and fluent transcripts significantly decreases the number of disfluencies in the output foot_12 . however, this is not sufficient for outperforming the baseline pipeline models on detecting and removing disfluencies, indicating that more complex architectures or mechanisms are required for effective end-toend asr and disfluency detection. the pipeline models have access to more information (i.e. the annotated disfluencies) than the end-to-end models; however, it is not clear if or how it would improve system performance. of the three end-to-end models, the transformer has the best performance on disfluency removal which we speculate is due to the self-attention mechanism which has been previously shown effective in detecting disfluencies in speech transcripts @xcite @xcite . we also compare the end-to-end asr and disfluency removal models with the pipeline asr and disfluency detection on the switchboard test set, as demonstrated in to further investigate the disfluency removal performance of the three end-to-end models, we randomly select 100 sentences from the switchboard dev set containing disfluencies. we categorize disfluencies into repetition, correction and restart according to the shriberg (1994) typology of speech repairs. repetitions are repairs where the reparandum and repair portions of the disfluency are identical, while corrections are where the reparandum and repairs differ (which are much harder to detect). restarts are where the speaker abandons a sentence and starts a new one (i.e. the repair is empty). as ## conclusion we showed wer is insufficient for evaluating endto-end asr and disfluency removal systems and alternatively introduced two metrics reflecting how well end-to-end systems handle disfluencies. we also showed the disfluency removal performance of end-to-end models is comparable to that of pipeline asr and specialized high-performance disfluency models. the best end-to-end system uses a transformer, that's what the best "off-the-shelf" disfluency detection system does, too. in the future, we aim to retrain the "off-the-shelf" disfluency detector on asr outputs using cross-validation. it is interesting to investigate how modifying the training loss would affect disfluency detection in end-to-end models. we also intend to augment the end-to-end transformer model with special mechanisms which have been previously shown effective for disfluency detection in speech transcripts.