Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disﬂuency Detection
Abstract
In modern interactive speech-based systems, speech is consumed and transcribed incrementally prior to having disﬂuencies removed.
This post-processing step is crucial for producing clean transcripts and high performance on downstream tasks (e.g. machine translation).
However, most current state-of-theart NLP models such as the Transformer operate non-incrementally, potentially causing unacceptable delays.
We propose a streaming BERT-based sequence tagging model that, combined with a novel training objective, is capable of detecting disﬂuencies in real-time while balancing accuracy and latency.
This is accomplished by training the model to decide whether to immediately output a prediction for the current input or to wait for further context.
Essentially, the model learns to dynamically size its lookahead window.
Our results demonstrate that our model produces comparably accurate predictions and does so sooner than our baselines, with lower ﬂicker.
Furthermore, the model attains state-of-the-art latency and stability scores when compared with recent work on incremental disﬂuency detection.
Introduction
Many modern Natural Language Understanding (NLU) applications (e g. transcribers, digital voice assistants, and chatbots) use streaming Automatic
Speech Recognition (ASR) systems that incrementally consume speech, offering real-time transcription and predictions with minimal delay.
However, these systems are often challenged by the presence of disﬂuencies, which are unintentional speech disruptions such as “um”, “no I meant”, and “I I I
I think,” that occur naturally in spontaneous speech.
Disﬂuencies not only hurt the readability of ASR transcripts, but also erode model performance on downstream tasks, such as machine translation (Hassan et al., 2014) and question answering (Gupta et al., 2021).
Indeed, even state-of-the-art models such as BERT (Devlin et al., 2019) and
T5 (Raffel et al., 2020) exhibit signiﬁcant drops in performance (as much as 28 and 20 F1 points, respectively) on the SQuAD-v2 question-answering benchmark (Rajpurkar et al., 2018) when disﬂuencies are inserted into the questions (Gupta et al., 2021).
Past work has shown that a prohibitively large amount of data is needed to train an end-toend dialogue model that is robust to the presence of disﬂuencies (Shalyminov et al., 2017).
As a result, modern ASR pipelines typically contain a separate post-processing step that detects and removes disﬂuencies from the transcript, which has been shown to perform better than end-to-end ASR models that generate ﬂuent text from disﬂuent speech (Jamshid Lou and Johnson, 2020).
Shriberg et al.
(1997) introduced the following disﬂuency schema components that are widely used in disﬂuency detection research: the reparandum (spoken segment intended to be removed), the interruption point (marked as “+”), the repair (spoken segment that comes as a replacement to the reparandum, of which the ﬁrst word is known as the repair onset ), and the interregnum (material that appears between the reparandum and repair).
An example of this annotation schema is shown in Figure 1.
Usually the disﬂuency detection task involves identifying and removing the reparandum portion of the disﬂuency.
One of the most popular approaches that targets disﬂuency detection is the usage of sequence tagging models such as ﬁnetuned BERT (Bach and Huang, 2019; Rohanian and Hough, 2021) or LSTM (Zayats et al., 2016; Rohanian and Hough, 2020).
Another challenge in disﬂuency detection is the fact that most interactive speech- or text-based applications consume input incrementally, producing predictions one word at a time, rather than in entire sentences.
However, recent state-of-the-art pretrained language models such as BERT have largely
A uh flight [ to Boston (cid:124) (cid:123)(cid:122) (cid:125) reparandum + { uh I mean (cid:124) (cid:123)(cid:122) (cid:125) interregnum } to (cid:124)(cid:123)(cid:122)(cid:125) [Figure]
onset (cid:124) (cid:123)(cid:122) (cid:125) [Figure] on been designed for non-incremental processing and are trained only to output predictions on complete input utterances.
Using a non-incremental model in an interactive setting produces undesirable delays, since downstream applications must wait for the user to ﬁnish their entire utterance before making any decisions.
To address the goal of streaming disﬂuency detection, recent work has focused on adapting non-incremental models for streaming settings.
Madureira and Schlangen (2020) demonstrated that
BERT-based models can adequately process incremental input for a variety of sequence tagging tasks when trained on partial sequences, although performance on full sequences suffers.
Rohanian and Hough (2021) applied both the truncated training and prophecy generation strategies from (Madureira and Schlangen, 2020) to a BERT LARGE model, achieving state-of-the-art performance on streaming metrics among incremental systems.
Notably, both these approaches employ the delay strategy of a ﬁxed lookahead window - a short amount of right context that the model can “peek” at when making its prediction on the current token (Buß and Schlangen, 2011).
Although a larger lookahead window can boost accuracy and stability, it also incurs extra delay (by deﬁnition).
In the task of incremental disﬂuency detection, a lookahead window is likely most useful for reparanda, since it is often nearly impossible to identify a reparandum without knowing whether it is followed by an interregnum or repair.
However, this extra right context may be much less informative for ﬂuent tokens.
Guided by this insight, we extend the past research by training a BERT-based model to dynamically decide how much lookahead context to use.
For each new input token that the model consumes, the model can choose to either immediately output a label for that token or to wait for further input before making its prediction.
We also design a novel training objective that accounts for both the cross-entropy and latency costs incurred by delaying inference.
In our experiments we explore the trade-offs between accuracy, latency, and output stability for both partial and complete results.
To our knowledge, this is the ﬁrst work to adapt the BERT architecture and training objective to balance accuracy and latency in a streaming sequence tagging task.
The contributions of our paper are as follows:
ﬁrst, we propose a new model architecture and training objective for streaming sequence tagging tasks.
This method involves ﬁne-tuning a pretrained BERT model to decide when to immediately output predictions and when to wait for further input– temporarily abstaining from producing a prediction.
Secondly, we show that this model achieves high accuracy in incremental settings with state-of-the-art latency and stability, all with a model architecture that is ∼ 35 times smaller than
BERT BASE (Zhao et al., 2021).
We demonstrate that the model continues to perform competitively in non-incremental settings when compared to its non-incremental counterparts.
Finally, our analyses show that our streaming model learns to wait the most when it encounters an interregnum or reparandum, and the least for ﬂuent or edit terms.
Related Work
Although disﬂuency detection itself is a wellstudied task, only a handful of past work has explored disﬂuency detection in an online settingthat is, consuming the input a single token at a time and outputting predictions on the partial input as early as possible.
Among the neural approaches, Hough and Schlangen (2015) were the ﬁrst to demonstrate competitive performance of recurrent neural networks (RNNs) on incremental disﬂuency detection by applying an Elman RNN paired with a Markov decoder that jointly optimized the probability of the output tag over the past inputs and outputs.
Hough and Schlangen (2017) built upon this by jointly training LSTMs on both utterance segmentation and disﬂuency detection, demonstrating that jointly training on the two tasks yielded higher accuracy and lower latency on both tasks than training on either task alone.
This was followed by a number of other works that also suc- cessfully paired incremental disﬂuency detection with other tasks, such as language modeling (Shalyminov et al., 2018) and POS tagging Rohanian and Hough (2020).
More recently, large pre-trained transformer architectures have demonstrated incredible success on sequence labeling tasks (Vaswani et al., 2017).
Although the original transformer architecture was not designed for streaming input, Chen et al.
(2020) proposed the controllable time-delay transformer instead, which combines a fast decoding strategy with a modiﬁed self-attention mechanism that attends only to future inputs in a ﬁxed lookahead window.
The closest work to ours is that of Rohanian and
Hough (2021), in which the authors ﬁne-tuned a pre-trained BERT LARGE model via add- M training, which feeds the model successive preﬁxes of lengths N + M, N + 2 M, · · · for each full-length training example.
Their best-performing model also made use of a prophecy decoding strategy, in which a GPT-2 (Radford et al., 2019) model predicted the missing right context of each partial input.
The BERT model then made its predictions based on the complete extrapolated sequence, POS tags, and word timings.
Unlike their work, we aim to train a more lightweight small vocabulary 12 × 128 BERT model that is more suitable for on-device settings, does not require a separate prophecy generation model, and uses only disﬂuency-annotated training data.
We also train our model on successive preﬁxes of the input (with N = 1 , M = 1 ) but modify both the architecture and training objective in order to balance the competing objectives of accuracy and latency.
Training a Streaming BERT Model
In this section we describe the architectural changes, new training objective, and training scheme that we use to adapt a (non-incremental)
BERT model for streaming sequence tagging tasks.
Speciﬁcally, we modify the model to enable it to either immediately produce a prediction for a given token or to decide to wait for further input.
These changes include a novel training objective that balances the cost between accuracy and latency – preventing the model from the extremes of either relying too much on waiting for further input or on speedily making predictions at the cost of accuracy.
We train this model using a restart-incremental training procedure described in Section 3.2.
3. Model Design
Streaming settings force models to make trade-offs between accuracy and latency.
More accurate predictions can be obtained by providing longer right context at the cost of incurring additional latency.
However, since most tokens are not disﬂuent, a model may not require the right context in order to accurately classify ﬂuent tokens.
Rather than using a ﬁxed lookahead window, we train a BERT model to jointly classify tokens and simultaneously choose the lookahead window size dynamically at each token.
Our proposed model architecture consists of a pre-trained BERT model with two separate token classiﬁcation heads added on top, as shown in Figure 2.
Each classiﬁcation head consists of a linear layer applied to the hidden layer outputs of the
BERT model.
The ﬁrst classiﬁcation head, the disﬂuency classiﬁer, is trained to classify whether each token is disﬂuent or not.
The second classiﬁcation head (the wait classiﬁer ) is trained to classify whether the model should wait for further input (temporarily abstain from predicting) or immediately output a prediction for the given token.
In effect, the wait classiﬁer decides how large of a lookahead window the model needs to make its prediction on the current input.
At inference time, we only output tokens that lie to the left of the ﬁrst token for which the model outputs a wait prediction and that are predicted to be ﬂuent.
This avoids potentially producing disjoint output in the case where the model produces predictions of wait followed by predict , making the output more clear for the user’s display.
We also adapt the training objective such that it accounts for both the accuracy and latency of the model’s outputs on each successive preﬁx.
Let ( x, y ) be the pair of input sequence and target output sequence with preﬁxes x 1 , x 2 , · · · , x | x | and y 1 , y 2 , · · · , y | x | , respectively where | x | is the length of the full sentence.
We also denote f ( x ) as the output logits of the disﬂuency classiﬁer, g ( x ) as the output logits of the wait classiﬁer, σ ( · ) as the softmax function, and H ( · , · ) as the cross-entropy loss.
Then the traditional cross-entropy loss on the full input and target sequences is
[Equation]
However, for each preﬁx we wish to only compute the cross-entropy loss on the tokens to the left of the ﬁrst token for which the model outputs a wait prediction.
To accomplish this, we devise a binary mask that zeros out the loss on the tokens to the right of and including the ﬁrst wait prediction:
[Equation]
where m
[Equation]
=
[Equation]
if j < k otherwise (
[Equation]
k =
[Equation]
where g ( x i )
j is the j -th element of vector g ( x i ) .
We then apply this mask to the cross-entropy loss for each preﬁx of example x to obtain preﬁx loss:
[Equation]
where we abuse notation here by denoting
H ( σ ( f ( x i )) , y i ) as the vector for which the jth element corresponds to the cross-entropy of ( σ ( f ( x i ))
j , y i,j ) and ◦ is element-wise multiplication.
Lastly, we deﬁne a latency cost that scales with both the probability of abstaining from classifying the j -th token in the i -th preﬁx ( σ ( g ( x i ) j ) ) and with the expected wait time, as measured by number of tokens, incurred by abstaining starting from token j in preﬁx x i :
[Equation]
Putting these together, the total loss for a single example ( x, y ) is:
[Equation]
with hyperparameters γ and λ controlling the relative strengths of the preﬁx and latency costs, respectively.
We also include the cross-entropy loss on the full sequences ( (cid:96) F ULL ) in addition to the preﬁx losses ( (cid:96) P REFIX ) because we wish for the model to maintain its ability to make predictions on full sequences.
Since g ( x ) does not appear anywhere in (cid:96) F ULL , the model is effectively forced to make predictions once it receives the full utterance.
Similarly, the (cid:96) L ATENCY term is essential because without it, the model could achieve minimal loss by always waiting (e.g. σ ( g ( x i ) j )
= 1 for all preﬁxes i and time steps j ), and only learning to classify disﬂuent tokens after receiving the full sequence.
This is equivalent to the non-incremental classiﬁcation loss.
If we instead set σ ( g ( x i ) j ) = 0 for all i, j (the case where the model never waits), the resulting loss is equivalent to the learning objective for strongly incremental training (see Section 3.2).
In essence, our training objective is a generalization of the strongly incremental objective.
3. Restart-Incremental Training
Although BERT models are typically ﬁne-tuned using complete pre-segmented sequences, an incremental model must process partial inputs at inference time, resulting in a distributional shift between the complete utterances typically seen in training datasets and the partial utterances seen at inference time.
A simple solution is to ﬁne-tune
BERT both on complete and partial inputs, a training scheme known as restart incrementality (Kahardipraja et al., 2021).
By providing successively extended preﬁxes of a given utterance to the model and computing the loss on the model outputs for each preﬁx, we can mimic the streaming data that the model would encounter in real time.
In all of our experiments, each successive preﬁx adds a single word to the previous preﬁx, a setting known as strongly incremental (Shalyminov et al., 2018).
Although this approach requires re-computation of the model outputs for each successive preﬁx, this also enables the model to correct its previous predictions, or to switch between waiting and predicting when it receives helpful right context.
Incorporating preﬁxes during training in incremental disﬂuency detection has been previously explored by Rohanian and Hough (2021).
This serves as a strong baseline in our experiments.
Experimental Setup
We ﬁne-tune all models on the Switchboard dataset (Godfrey et al., 1992), a transcribed English multispeaker conversational corpus that is commonly used for ASR research.
We speciﬁcally use the version from the Linguistic Data Corpus’s Treebank- (Marcus et al., 1999) distribution, which additionally contains disﬂuency annotations and a standard train/dev/test split (Charniak and Johnson, 2001).
We follow Rocholl et al.
(2021), training our models to classify both the reparanda and interregna as disﬂuent for future removal in a ﬁnal post-processed transcript.
4. Baselines
All of our experiments use small distilled BERT models, speciﬁcally a small vocabulary BERT model (Zhao et al., 2021) (BERT SV ) with 12 hidden layers of size 128 that is pre-trained on English
Wikipedia and BookCorpus (Zhu et al., 2015).
The details of our hyperparameter tuning can be found in the Appendix (Section A.2).
We use small models for two reasons: 1) ﬁnetuning a model on all given preﬁxes of each training example is resource intensive, and 2) many streaming natural language understanding applications run entirely on mobile devices which precludes the use of large models.
Previous work on small non-incremental BERT-based models used for disﬂuency detection (Rocholl et al., 2021) showed signiﬁcant improvement in memory and latency without compromising task performance.
The core
BERT SV model is a distilled version of BERT BASE with smaller vocabulary and reduced hidden layer dimensions (Zhao et al., 2021).
Due to its smaller vocabulary size (5K versus 30K tokens), the model has only about 3.1M parameters, as compared to
BERT BASE ’s approximately 108.9M parameters, achieving around 80% latency reduction.
In order to isolate the effects of training with restart incrementality (Section 3.2) versus the improvements derived directly from incorporating our new training objective, we also evaluate two other models: 1) a non-incremental BERT SV model trained in the usual way, on full sequences; and 2) a BERT SV model trained with restart incrementality - i.e., on all preﬁxes of every training example (which we will refer to as “all preﬁxes” in following tables).
Setup (1) is equivalent to ablating both (cid:96) P REFIX and (cid:96) L ATENCY whereas setup (2) is equivalent to ablating only (cid:96) L ATENCY .
We do not ablate (cid:96) P REFIX in isolation since this leaves only the (cid:96) F ULL and (cid:96) L ATENCY terms, and there does not exist a meaningful measure of latency when the model never needs to wait for more input (since it is always given the full utterance as input).
For each of these baseline models we also follow Rohanian and Hough (2021) and Kahardipraja et al.
(2021) by evaluating with different ﬁxed lookahead (LA) window sizes of LA = 0 , 1 , 2 .
4. Incremental Evaluation
Accuracy alone is not a sufﬁcient measure of success to robustly evaluate a streaming model.
Since a streaming model is meant to operate in real time, it should return output as soon as possible after it receives new input.
As such, we also need to evaluate it with respect to latency – i.e. the number of new tokens a model must consume before producing a prediction for the current token.
Furthermore, streaming models are often designed to be capable of retroactively changing their predictions on previous tokens as new input arrives.
This introduces the risk of output “jitter” or “ﬂicker,” where the output changes dramatically as new input is consumed, necessitating evaluation of stability.
To capture all these important dimensions of streaming model performance, we evaluate the models using the following diachronic metrics (with formulas and further details in the Appendix):
• Streaming F 1 : An accuracy metric scored in the same way as the typical F 1 score, albeit we score the predictions for a single token over the course of multiple time steps separately as if they were predictions for separate tokens.
[Table] [Table]
• Edit Overhead (EO) (Buß and Schlangen, 2011):
A stability metric that measures the average number of unnecessary edits, normalized by utterance length.
• Time-to-detection (TTD) (Hough and Schlangen, 2017):
A latency metric that is only computed on disﬂuent tokens that are classiﬁed correctly.
It is the average amount of time (in number of tokens consumed) that the model requires before ﬁrst detecting a disﬂuency.
As mentioned earlier, we include both reparanda and interregna as disﬂuencies.
• Average waiting time (AWT) :
The average amount of time (in number of tokens consumed) that the model waits for further input before making a prediction on a given token.
For models with a ﬁxed lookahead window, this is equivalent to the lookahead window size.
For the streaming model, this is equivalent to the average lookahead window size.
• First time to detection (FTD) (Zwarts et al., 2010; Rohanian and Hough, 2021):
Similar to to the TDD metric described above with the main difference being that the latency (in number of words) is calculated starting from the onset of a gold standard repair.
Results
In this section we present a summary of both the non-incremental and incremental performance of our streaming model against that the baselines.
We also present an analysis of the types of errors and average amount of waiting time the streaming model incurs.
5. Incremental Performance
Table 1 shows both the incremental and nonincremental evaluation metrics.
Our proposed streaming BERT SV model achieved a 9% increasein streaming F 1 over both of the baselines (with lookahead = 0), as well as a 71% and 72% decrease in edit overhead compared to the non-streaming models trained on full sequences and all preﬁxes, respectively.
Despite being trained with a different architecture and loss objective, the streaming model does not sacriﬁce its non-incremental performance, yielding a ﬁnal output F 1 score that is only one point less than its non-streaming counterparts.
Generally speaking, when the streaming model does output a prediction, it classiﬁes tokens as disﬂuent less often than the non-streaming models with zero LA window, achieving much higher precision (P) and marginally lower recall (R), resulting in a model that “ﬂickers" less frequently.
However, this does contribute to a slightly higher time-to-detection score compared to the baselines with zero lookahead, since the streaming model is generally less aggressive but more precise with outputting disﬂuent predictions.
When compared to the models with ﬁxed lookahead (the lower half of Table 1), however, the streaming model always achieves lower TTD while achieving signiﬁcantly lower waiting time and comparable accuracy and stability.
[Table]
Effect of lookahead window size
We also evaluated the performance of the non-streaming baseline models with ﬁxed lookahead window sizes of 1 and 2 tokens, as shown in the lower half of
Table 1.
In line with what has been reported in past work (Madureira and Schlangen, 2020; Buß and Schlangen, 2011), the size of the lookahead window scales directly with the accuracy and stability and inversely with the latency of the model.
However, the streaming model has comparable streaming F 1 and edit overhead scores as the nonstreaming models with LA = 1 , even though the streaming model has 79% less average wait time.
This indicates that the streaming model is able to correctly classify tokens sooner and with more stability than the baseline models that have
LA = 1 .
Although the models with LA = 2 improve marginally on accuracy and stability over the models with LA = 1 , the streaming model continues to have lower TTD and AWT but comparable F 1 and EO when compared to the models with LA = 2 .
The utility of dynamic lookahead
The results in Table 1 also reveal some insights into which parts of the model design and training scheme are more important for streaming performance and efﬁciency.
Merely training a non-streaming model on preﬁxes of the training examples appears to have minimal effect on F 1 , precision, and recall, but does somewhat improve the TTD score.
We hypothesize that this is largely the result of training on a data distribution that more closely resembles the test distribution.
Adding the extra wait classiﬁer head and latency cost term in the training objective yields the greatest improvements in both precision and stability, as seen in the differences in
F 1 , P, and EO values between the BERT SV model trained on all preﬁxes and the streaming BERT SV model.
When to wait
Since the streaming model can abstain from outputting predictions for arbitrarily long sufﬁxes of the input, it incurs waiting time - an average of 0.21 tokens more than the non-streaming models with 0 lookahead.
Table 2 shows that the streaming model abstains the most when encountering interregna and reparanda, waiting for approximately 1.06 and 0.76 more tokens, respectively.
Given that it is easier to identify a disﬂuency once the entire reparandum and interregnum have been observed, it follows that the model’s predictions may be more uncertain for reparanda and interregna upon ﬁrst consumption, thus incurring the highest average waiting times.
An example of the model’s incremental outputs for a disﬂuency structure is shown in Table 4.
For correctly classiﬁed disﬂuent tokens, the streaming model also has a higher TTD, likely because the non-incremental models are more aggressive in predicting disﬂuent labels (while making more errors) than the streaming model.
Still, this TTD is lower than all the models with ﬁxed lookahead windows.
5. Error Analysis
Figure 3 shows an error analysis on the models’ predictions.
We computed the percentage of the
[Table]
time that the streaming model misclassiﬁed a token, counting each incidence of the token across each time step separately.
All models achieved the lowest misclassiﬁcation rates on ﬂuent and edit tokens, and the highest misclassiﬁcation rates on reparanda tokens.
For tokens that were ﬂuent, edit terms, or part of a repair or repair onset, the streaming model achieved signiﬁcantly lower misclassiﬁcation rates than the baselines.
However, all three models performed comparably on interregna and reparanda.
Since we measure misclassiﬁcation rate on every token at every time step, the high misclassiﬁcation rates on reparanda are expected as it is often not feasible to detect a reparandum until an interregnum or repair onset has been seen.
Accuracy versus latency
In comparison to the baseline models with 0 lookahead, the streaming model makes the largest tradeoffs in accuracy versus latency for repair onsets and repairs, as shown by Figure 3 and Table 2.
While the streaming model incurs average additional wait times of 0.74 and 0.46 tokens for repairs and repair onsets respectively, its misclassiﬁcation rates are also approximately 85% and 82% less than the baseline models on repairs and repair onsets respectively.
In addition, Table 1 demonstrates that the streaming model still achieves comparable F 1 and greater stability (lower EO) in comparison to the non-streaming baselines with lookahead 1, despite having an average wait time that is 79% shorter.
5. Comparison with Competitor Baselines
As shown in Table 3, in comparison with the
BERT LARGE -based prophecy decoding model proposed in Rohanian and Hough (2021), our streaming model achieves state-of-the-art stability (85% decrease in EO) and latency (65% decrease in
FTD) 2 , despite having far fewer parameters.
Conclusion and Future Work
We have introduced a streaming BERT-based Transformer model that is capable of balancing accuracy with latency by simultaneously making token-level disﬂuency predictions and dynamically deciding how large of a lookahead window to use.
Our approach improves both streaming accuracy and output stability on an incremental disﬂuency detection task.
Furthermore, it incurs very low average latency in comparison with non-incremental
BERT models of the same size.
Lastly, our model requires minimal lookahead beyond disﬂuent regions and achieves state-of-the-art edit overhead and ﬁrst-time-to-detection scores compared to past work (Rohanian and Hough, 2021).
While the main focus of this paper has been on developing a fast, accurate, and stable streaming model for disﬂuency detection, our approach is general enough to be used in other incremental tagging models of linguistic phenomena that beneﬁt from the right context for optimal accuracy.
In future work we are interested in applying this approach to tasks such as real-time punctuation prediction and incremental parsing.
Furthermore, the streaming model’s efﬁciency is limited by its non-autoregressive nature and training via restart incrementality.
Future work should also explore how to apply a dynamic lookahead window without re-computing the predictions on all the previous inputs and to rely on fewer preﬁxes during training. presented
Ethical Considerations
While our work does not introduce a new dataset, it does depend on a training dataset that was collected from ﬂuent English-speaking, able-bodied human subjects.
If deployed in a real-world application, this model would likely perform noticeably worse for users who speak with non-American accents or speech impediments.
Transcripts for these users could be disproportionately noisy and the streaming model’s average wait time would likely also be longer.
Care should be taken to assess the sensitivity and robustness of such a model to non-ﬂuent or non-American English prior to deployment.
This model should also be used very cautiously in situations where mistakenly eliding ﬂuent portions of speech from the captions or transcript could incur dire consequences, such as in an emergency call center.  Acknowledgments
We thank Johann Rocholl, Colin Cherry, Dan
Liebling, Noah Murad, and members of the Google
Research Speech Intelligence team for valuable feedback and discussion.
We also thank the Google Student Researcher and research internship programs for supporting this work. Lastly, we also thank the anonymous reviewers for their thorough and helpful comments. A Appendix A. Evaluation Metrics
We provide more detailed formulas here for each of our evaluation metrics.
For a given input utterance x , let x [ j ] be the j -th token of x , y
[ j ] be the gold label (either 0 for ﬂuent or 1 for disﬂuent) for x [ j ] , x [: i ] be the i -th preﬁx of x (i.e. the ﬁrst i tokens of x ), f ( x [: i ])
[ j ] be the predicted label for x [ j ] after the model has consumed preﬁx i , D be the entire dataset that we are evaluating predictive performance for, and | D | be the size of D .
Accuracy metrics
Our deﬁnitions of the streaming true positives, true negatives, false positives, and false negatives are:
T P streaming =
|{ ( i, j ) | y
[ j ] = 1 , f ( x [: i ])
[ j ] = 1 }|
T N streaming =
|{ ( i, j ) | y
[ j ] = 0 , f
( x [: i ])
[ j ] = 0 }|
F P streaming =
|{ ( i, j ) | y
[ j ] = 0 , f
( x [: i ])
[ j ] = 1 }|
F N streaming =
|{ ( i, j ) | y
[ j ] = 1 , f ( x [: i ])
[ j ] = 0 }|
Similar to the traditional deﬁnitions, streaming precision, recall, and F 1 are computed as:
[Equation]
Stability metrics
• Edit overhead (EO) (Buß and Schlangen, 2011):
To calculate edit overhead, we need to ﬁrst identify, for each preﬁx x
[: i ] of a given input x , which tokens x [ j ] have predictions f ( x [: i ])
[ j ] that differ from the model’s pre- diction in the previous preﬁx f ( x [: i − 1])
[ j ] .
Denoting the cardinality of this set as E ( x ) (for the number of edits the model makes on x ), we have:
[Equation]
Then we can compute EO as follows:
[Equation]
where | x | is the number of tokens in x .
Latency metrics
• Time-to-detection
(TTD) (Hough and
Schlangen, 2017): Since time-to-detection (TTD) is measured only on disﬂuent tokens that are eventually predicted as such by the model, we need to ﬁrst deﬁne the set of tokens in a given example x that are true positives at some point:
[Equation]
Then for a given token x [ k ] ∈ x , the detection time (DT) can be calculated as:
[Equation]
It follows that the TTD for the entire dataset
D is the average DT for all disﬂuent tokens that are eventually detected for all x ∈ D :
[Equation]
where
[Equation]
the total number of disﬂuent
tokens in the dataset that are eventually detected by the model.
• First time to detection (FTD) (Zwarts et al., 2010; Rohanian and Hough, 2021): Similar to TTD, this metric is only measured on dis-
ﬂuent tokens that are eventually detected by the model.
Then given some x [ k ] ∈ T P ( x ) , let RI ( x [ k ]) represent the index of the ﬁrst token in the repair that follows x [ k ] .
Since we are measuring detection time from the start of a gold standard repair instead, the detection time becomes:
[Equation]
The rest of the formula for the FTD is similar to that of the TTD:
[Equation]
where
[Equation]
the total number of disﬂuent tokens in the dataset that are eventually detected by the model.
• Average waiting time (AWT) : Suppose that given an input token x [ k ] , the model can abstain from making a prediction (which occurs both with the streaming model and with the ﬁxed lookahead models).
We denote this outcome as y [ k ] = ∅ .
To compute AWT, we ﬁrst calculate the ﬁrst prediction time (FPT) for a given token x [ k ] ,
[Equation]
i.e.
the
ﬁrst time step i in which the model outputs a prediction for token x [ k ] .
Then the AWT is
[Equation]
where
[Equation]
the total number of tokens in the dataset.
A. Model Training and Hyperparameter Tuning
We implemented our models using TensorFlow v2.7 (Abadi et al., 2015) and the Hugging Face transformers library (Wolf et al., 2020).
We also ﬁne-tuned all model hyperparameters using Vizier (Golovin et al., 2017), a black-box optimization system, using streaming F1 score on the Switchboard validation set as our objective.
The searched ranges for each hyperparameter were learning rate ∈ [1 × 10 − 5 , 1 × 10 − 1 ] , number of training epochs ∈ [12 , 20] , λ ∈ [1 × 10 − 8 , 1 × 10 − 6 ] , γ ∈ [1 , 10] .
For most experiments we ran 30 trials total, with 10 evaluations in parallel.
Each individual trial (one set of hyper-parameters) ran on a single NVIDIA P100 GPU.
Experimental run time varied from about 13 to 24 hours, depending mostly on the number of epochs.
For each model variant we present only the results from the conﬁguration with the highest streaming F 1 score on the Switchboard validation dataset.
Our best performing streaming model used parameter values of λ = 1 . 5 × 10 − 7 , learning rate 1 . 2 × 10 − 4 , γ = 1 . 9 , training batch size 8, and 12 epochs.
[Table]