Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection (2022)

Abstract: In modern interactive speech-based systems, speech is consumed and transcribed incrementally prior to having disfluencies removed. While this post-processing step is crucial for producing clean transcripts and high performance on downstream tasks (e.g. machine translation), most current state-of-the-art NLP models such as the Transformer operate non-incrementally, potentially causing unacceptable delays for the user. In this work we propose a streaming BERT-based sequence tagging model that, combined with a novel training objective, is capable of detecting disfluencies in real-time while balancing accuracy and latency. This is accomplished by training the model to decide whether to immediately output a prediction for the current input or to wait for further context, in essence learning to dynamically size the lookahead window. Our results demonstrate that our model produces comparably accurate predictions and does so sooner than our baselines, with lower flicker. Furthermore, the model attains state-of-the-art latency and stability scores when compared with recent work on incremental disfluency detection.

## introduction many modern natural language understanding (nlu) applications (e.g. transcribers, digital voice assistants, and chatbots) use streaming automatic speech recognition (asr) systems that incrementally consume speech, offering real-time transcription and predictions with minimal delay. however, these systems are often challenged by the presence of disfluencies, which are unintentional speech disruptions such as "um", "no i meant", and "i i i i think," that occur naturally in spontaneous speech. disfluencies not only hurt the readability of asr transcripts, but also erode model performance on downstream tasks, such as machine translation @xcite and question answering @xcite . indeed, even state-of-the-art models such as bert @xcite and t5 @xcite exhibit significant drops in performance (as much as 28 and 20 f1 points, respectively) on the squad-v2 question-answering benchmark @xcite when disfluencies are inserted into the questions @xcite . past work has shown that a prohibitively large amount of data is needed to train an end-toend dialogue model that is robust to the presence of disfluencies @xcite . as a result, modern asr pipelines typically contain a separate post-processing step that detects and removes disfluencies from the transcript, which has been shown to perform better than end-to-end asr models that generate fluent text from disfluent speech @xcite . @xcite introduced the following disfluency schema components that are widely used in disfluency detection research: the reparandum (spoken segment intended to be removed), the interruption point (marked as "+"), the repair (spoken segment that comes as a replacement to the reparandum, of which the first word is known as the repair onset), and the interregnum (material that appears between the reparandum and repair). an example of this annotation schema is shown in figure 1 . usually the disfluency detection task involves identifying and removing the reparandum portion of the disfluency. one of the most popular approaches that targets disfluency detection is the usage of sequence tagging models such as finetuned bert @xcite or lstm @xcite . another challenge in disfluency detection is the fact that most interactive speech-or text-based applications consume input incrementally, producing predictions one word at a time, rather than in entire sentences. however, recent state-of-the-art pretrained language models such as bert have largely been designed for non-incremental processing and are trained only to output predictions on complete input utterances. using a non-incremental model in an interactive setting produces undesirable delays, since downstream applications must wait for the user to finish their entire utterance before making any decisions. to address the goal of streaming disfluency detection, recent work has focused on adapting non-incremental models for streaming settings. @xcite demonstrated that bert-based models can adequately process incremental input for a variety of sequence tagging tasks when trained on partial sequences, although performance on full sequences suffers. rohanian and hough (2021) applied both the truncated training and prophecy generation strategies from @xcite to a bert large model, achieving state-of-the-art performance on streaming metrics among incremental systems. notably, both these approaches employ the delay strategy of a fixed lookahead window -a short amount of right context that the model can "peek" at when making its prediction on the current token @xcite . although a larger lookahead window can boost accuracy and stability, it also incurs extra delay (by definition). in the task of incremental disfluency detection, a lookahead window is likely most useful for reparanda, since it is often nearly impossible to identify a reparandum without knowing whether it is followed by an interregnum or repair. however, this extra right context may be much less informative for fluent tokens. guided by this insight, we extend the past research by training a bert-based model to dynamically decide how much lookahead context to use. for each new input token that the model consumes, the model can choose to either immediately output a label for that token or to wait for further input before making its prediction. we also design a novel training objective that accounts for both the cross-entropy and latency costs incurred by delaying inference. in our experiments we explore the trade-offs be-tween accuracy, latency, and output stability for both partial and complete results. to our knowledge, this is the first work to adapt the bert architecture and training objective to balance accuracy and latency in a streaming sequence tagging task. the contributions of our paper are as follows: first, we propose a new model architecture and training objective for streaming sequence tagging tasks. this method involves fine-tuning a pretrained bert model to decide when to immediately output predictions and when to wait for further input-temporarily abstaining from producing a prediction. secondly, we show that this model achieves high accuracy in incremental settings with state-of-the-art latency and stability, all with a model architecture that is ∼ 35 times smaller than bert base @xcite . we demonstrate that the model continues to perform competitively in non-incremental settings when compared to its non-incremental counterparts. finally, our analyses show that our streaming model learns to wait the most when it encounters an interregnum or reparandum, and the least for fluent or edit terms. ## related work although disfluency detection itself is a wellstudied task, only a handful of past work has explored disfluency detection in an online settingthat is, consuming the input a single token at a time and outputting predictions on the partial input as early as possible. among the neural approaches, @xcite were the first to demonstrate competitive performance of recurrent neural networks (rnns) on incremental disfluency detection by applying an elman rnn paired with a markov decoder that jointly optimized the probability of the output tag over the past inputs and outputs. @xcite built upon this by jointly training lstms on both utterance segmentation and disfluency detection, demonstrating that jointly training on the two tasks yielded higher accuracy and lower latency on both tasks than training on either task alone. this was followed by a number of other works that also successfully paired incremental disfluency detection with other tasks, such as language modeling @xcite and pos tagging @xcite . more recently, large pre-trained transformer architectures have demonstrated incredible success on sequence labeling tasks @xcite . although the original transformer architecture was not designed for streaming input, @xcite proposed the controllable time-delay transformer instead, which combines a fast decoding strategy with a modified self-attention mechanism that attends only to future inputs in a fixed lookahead window. the closest work to ours is that of @xcite , in which the authors fine-tuned a pre-trained bert large model via add-m training, which feeds the model successive prefixes of lengths n + m, n + 2m, • • • for each full-length training example. their best-performing model also made use of a prophecy decoding strategy, in which a gpt-2 @xcite model predicted the missing right context of each partial input. the bert model then made its predictions based on the complete extrapolated sequence, pos tags, and word timings. unlike their work, we aim to train a more lightweight small vocabulary 12 × 128 bert model that is more suitable for on-device settings, does not require a separate prophecy generation model, and uses only disfluency-annotated training data. we also train our model on successive prefixes of the input (with @xmath0, @xmath1. ## training a streaming bert model in this section we describe the architectural changes, new training objective, and training scheme that we use to adapt a (non-incremental) bert model for streaming sequence tagging tasks. specifically, we modify the model to enable it to either immediately produce a prediction for a given token or to decide to wait for further input. these changes include a novel training objective that balances the cost between accuracy and latency -preventing the model from the extremes of either relying too much on waiting for further input or on speedily making predictions at the cost of accuracy. we train this model using a restart-incremental training procedure described in section 3.2. ## experimental setup we fine-tune all models on the switchboard dataset @xcite , a transcribed english multispeaker conversational corpus that is commonly used for asr research. we specifically use the version from the linguistic data corpus's treebank-3 @xcite distribution, which additionally contains disfluency annotations and a standard train/dev/test split @xcite . we follow @xcite , training our models to classify both the reparanda and interregna as disfluent for future removal in a final post-processed transcript. ## results in this section we present a summary of both the non-incremental and incremental performance of our streaming model against that the baselines. we also present an analysis of the types of errors and average amount of waiting time the streaming model incurs. ## conclusion and future work we have introduced a streaming bert-based transformer model that is capable of balancing accuracy with latency by simultaneously making token-level disfluency predictions and dynamically deciding how large of a lookahead window to use. our approach improves both streaming accuracy and output stability on an incremental disfluency detection task. furthermore, it incurs very low average latency in comparison with non-incremental bert models of the same size. lastly, our model requires minimal lookahead beyond disfluent regions and achieves state-of-the-art edit overhead and first-time-to-detection scores compared to past work @xcite . while the main focus of this paper has been on developing a fast, accurate, and stable streaming model for disfluency detection, our approach is general enough to be used in other incremental tagging models of linguistic phenomena that benefit from the right context for optimal accuracy. in future work we are interested in applying this approach to tasks such as real-time punctuation prediction and incremental parsing. furthermore, the streaming model's efficiency is limited by its non-autoregressive nature and training via restart incrementality. future work should also explore how to apply a dynamic lookahead window without re-computing the predictions on all the previous inputs and to rely on fewer prefixes during training.