Best of Both Worlds: Making High Accuracy Non-incremental Transformer-based Disfluency Detection Incremental (2021)

Abstract: While Transformer-based text classifiers pre-trained on large volumes of text have yielded significant improvements on a wide range of computational linguistics tasks, their implementations have been unsuitable for live incremental processing thus far, operating only on the level of complete sentence inputs. We address the challenge of introducing methods for word-by-word left-to-right incremental processing to Transformers such as BERT, models without an intrinsic sense of linear order. We modify the training method and live decoding of non-incremental models to detect speech disfluencies with minimum latency and without pre-segmentation of dialogue acts. We experiment with several decoding methods to predict the rightward context of the word currently being processed using a GPT-2 language model and apply a BERT-based disfluency detector to sequences, including predicted words. We show our method of incrementalising Transformers maintains most of their high non-incremental performance while operating strictly incrementally. We also evaluate our models’ incremental performance to establish the trade-off between incremental performance and final performance, using different prediction strategies. We apply our system to incremental speech recognition results as they arrive into a live system and achieve state-of-the-art results in this setting.

## introduction conversational systems provide a significant addition to the present approaches in mental health care delivery. interactions with these conversational agents have been shown to contain observable indicators of cognitive states, such as the rate of filled pauses and different temporal and turnrelated features @xcite . alzheimer's disease (ad) patients, for example, have trouble performing tasks that leverage semantic information; they have difficulties with verbal fluency and object recognition. ad patients speak more slowly with long pauses and spend extra time looking for the correct word, which leads to speech disfluency @xcite . disfluency markers can be key features for identifying certain cognitive disorders for application in conversational agents @xcite . such conversational systems are primarily used for content processing, which is then analyzed offline. there is much work on detecting disfluencies for offline analysis of transcripts. however, given that these disfluency detection models do not work for live systems and depend on rich transcription data, including pre-segmentation of dialogue acts, to facilitate more cost-effective analysis of other data, we need systems capable of performing directly and incrementally off the speech signal, or at least from the results of automatic speech recognition (asr) as they arrive in the system. as it receives word-by-word data, an incremental model must operate with minimum latency and do so without changing its initial assumptions and delivering its best decisions as early as possible following the principles outlined in @xcite . here we design and evaluating models that work with online, incremental speech recognition output to detect disfluencies with varying levels of granularity. the best neural language encoders currently used in computational linguistics consider word sequences as a whole, and their implementations have been unsuitable for live incremental processing. transformers @xcite , for instance, operate on representations that do not naturally have an organizing principle of linear word order. we analyze how these models work under incremental frameworks, where it is essential to present partial output relying on partial input pro-vided up to a certain time step that may occur in interactive healthcare systems. we explore whether we can adjust such models to function incrementally and how useful they are in terms of overall accuracy and incremental metrics. to further enhance the models' incremental performance, we use two general strategies to adjust the training regime and the real-time procedure: incremental training ('chunk-based' training and add-m training) and incremental decoding (constant latency and prophecies). we employ three prominent decoding methods to predict the rightward context of the word currently being processed: beam search, top-k sampling, and top-p sampling. we also measure our models' incremental performance to set the trade-off between incremental performance and final performance. ## related work although considerable work has been done on detecting disfluencies, much of this work uses transcripts as texts rather than live speech inputs, with the goal of 'cleaning' the disfluent content for post-processing purposes. they are almost exclusively conducted on pre-segmented utterances of the switchboard corpus of telephone conversations @xcite . several disfluency detection efforts involve sentence-based parsing and language models @xcite . sequence labeling models with start-inside-outside (bio) style tags have been used in recent neural sequence approaches to disfluency detection based on bi-directional long short term memory (bilstm) networks and transformers, in which the sequences are available in full @xcite @xcite . such offline methods are insufficient if we intend to infer meaning from repairs and edit words for disfluency detection in real-time, which is beneficial in a healthcare domain dialogue system that seeks to get a consistent and clear understanding of user statements and the user's cognitive state. methods based on strictly incremental operation have been rare. @xcite used a line of classifiers and language model features in a strong incremental operating system without looking ahead. incremental dependency parsing combined with the removal of disfluency was also studied @xcite . some studies have used recurrent neural networks for live dis-fluency identification. using a basic elman recurrent neural network (rnn), @xcite investigated incremental processing, with an objective coupling detection accuracy with low latency. language models have been used as an additional task for the identification of disfluencies, relying on the intuition that disfluencies can be detected by divergences from clean language models, with @xcite 's noisy channel model beginning this effort. @xcite made language modelling an auxiliary task to disfluency detection in a deep multi-task learning (mtl) set-up, gaining accuracy over a vanilla rnn tagger. pos tags have also been used as an input for detecting disfluencies, showing slight increases in disfluency detection over using word values alone @xcite . while the work above operates only on transcripts pre-segmented into utterances, recent research has been performed on combining disfluency detection with utterance segmentation. this was done in a joint tagset of disfluency, and utterance segmentation tags by @xcite , showing an improvement over the performance of the individual tasks, and @xcite show an improvement in both tasks when framed as a multi-task learning (mtl) set-up with a long short-term memory network (lstm), also simultaneously doing pos-tagging and language modelling. the recent live incremental systems fall short of the same accuracies achievable on pre-segmented transcripts, so there is a natural interest in using the best non-incremental sequence models and adapting them for incrementality. @xcite take up this effort in several other sequence tagging and classification tasks, showing how bidirectional encoders and transformers can be modified to work incrementally. to reduce the impact of the partiality of the input, the models predict future content and wait for more rightward context. @xcite also use truncated inputs during the training phase of live machine translation to address the partial input sentence decoding problem bidirectional encoders face. here, we seek to add to this growing effort to investigate the trade-off of incremental performance against the final output quality of deep neural network-based language processing, applied to incremental disfluency detection. ## disfluency detection disfluencies are generally assumed to have a reparandum-interregnum-repair structure in their fullest form as speech repairs @xcite . a reparandum is a stretch of speech later corrected by the speaker; the corrected expression is a repair, the beginning of which is referred to as repair onset. an interregnum word is a filler or a reference expression between the repair and reparandum, usually an interruption and hesitation step when the speaker expresses a repair, giving the structure as in (1). in the absence of reparandum and repair, the disfluency is reduced to an isolated edit term. a marked, lexicalised edit term such as a filled pause ("uh" or "um") or more phrasal terms such as "i mean" and "you know" may occur. the identification of these elements and their structure is then the task of disfluency detection. the task of detecting incremental disfluencies adds to the difficulty of doing this in real-time, word-by-word, from left to right. disfluency recognition is then treated as the same problem that a human processor faces with a disfluent expression: only when an interregnum is detected, or maybe even when a repair is initiated, does it become clear that the earlier content is now to be regarded as 'to be repaired,' i.e., to be classified as a reparandum. therefore, the task cannot be defined as a simple sequence labeling task in which the tags for the reparandum, interregnum, and repair phases are assigned left-to-right over words as seen in the above example; in this case, it will require the assumption that "likes" would be repaired, at a time when there is no data to make it available. we use a tag set that encodes the start of the reparandum only at a time when it can be inferred, primarily when the repair starts -the disfluency detection task is to tag words as in the top line of tags in fig. 1 as either fluent (f ) an edit term (e), a repair onset word (rps-n for the reparandum starting n words back) and a repair end word of the type repeat (rpnrep), substitution (rpnsub) or delete (rpndel). ## model to incrementalise a transformer-based model for word-by-word disfluency detection, we devise a model built on top of a pre-trained bert architecture @xcite with a conditional random field (crf) output architecture to tag sequences with tags such as those in the top line of fig. 1 . we use a bert-based encoder and try different strategies to incrementalise the system's operation and output, using language models to predict future word sequences as described in section 5 while maintaining bert's non-incremental quality. utterance segmentation our models are designed to work not only with pre-segmented data but also on raw transcripts and asr results, where utterance segmentation is required to leverage the use of sentence-based linguistic knowledge in bert. utterance segmentation has a clear interdependence with and influence on the detection of disfluency as disfluent restarts and repairs may be incorrectly predicted at fluent utterance boundaries without segmentation. in this paper, rather than performing utterance segmentation in tandem with disfluency detection, we perform it on words as they arrive in the system as a live segmentation task before sending the current prefix of the utterance to the disfluency detection system. we use the word-by-word segmentation system from (rohanian and hough, 2020) where four output tags define ranges of transcribed words or word hypotheses using a bies tag scheme (beginning, inside, end, and single) to allow for the prediction of an utterance ending. the tagset allows information to be captured from the context of the word to decide whether this word continues a current utterance (theprefix) or starts anew (the . prefix), and also allows live prediction of whether the next word will continue the current utterance (thesuffix) or whether the current word finishes the utterance (the . suffix). an example of the scheme is shown in the second line of fig. 1 . crf we use a crf output architecture to predict a tag for every token. although this model generates predictions for the whole sequence, the labels are outputted individually. there are important dependencies between adjacent labels in disfluency detection, and explicit modeling of these relationships can help. the addition of the crf enables the model to test for the most optimal path across all available label sequences. ## strategies for incrementalising bert here we describe the different strategies we used to modify the training and live decoding methods of non-incremental models to detect speech disfluencies word-by-word incrementally. the general principle is to leverage high accuracy full sequence classification using bert but deploying it on sequences, including future predictions for words up to the hypothesised end of the current utterance. ## experimental set-up we train on transcripts and test on both transcripts and asr hypotheses. all models in testing have strictly word-by-word left to right input. in addition to using the latest word hypothesis as input, we train and evaluate the presented models with two kinds of additional inputs: time elapsed from the end of the previous word (hypothesis) to the current one and the pos tag of the current word. results on the development set were used to find the best model to be evaluated on the test set. we used the data from @xcite for asr hypotheses -this was generated by a free trial version of ibm's watson speech-to-text service for incremental asr. the service offers good quality asr on noisy data-on our selected held-out data on switchboard, and the average wer is 26.5%. the watson service, crucially for our task, does not filter out hesitation markers or disfluencies @xcite . the service delivers results incrementally, so silence-based endpointing is not used. it also outputs word timings, which are close enough to the source timings to use as features in the live version of our system. the word embedding for lstm was initialised with 50-dimensional embedding trained on google news @xcite . the model has been implemented using tensorflow 2.1. we train all models for a maximum of 50 epochs; otherwise, stop training if there is no improvement on the best score on the validation set after 7 epochs. a large version of the pre-trained bert is used with 340m parameters (24-layer blocks, 16 selfattention heads, and 1024 hidden-size) for the model. in our analysis, when fine-tuning bert, we followed the hyper-parameters of @xcite . since the datasets we use are tokenized, and each token has a matching tag, we adopt the directions provided by @xcite to deal with the sub-tokenization of bert: to determine its label, the scores of the first sub-token are used, and further sub-token scores are discarded. data we use standard switchboard training data (all conversation numbers starting sw2*,sw3 * in the penn treebank iii release: 100k utterances, 650k words) and use standard held-out data (ptb iii files sw4[5-9] *: 6.4k utterances, 49k words) as our validation set. we test on the standard test data (ptb iii files 4[0-1] *) with partial words and punctuation stripped away from all files. we only choose a subset of the held-out and test data for the asr results in assessment, whereby both channels achieve below 40 percent wer to ensure good separation-this left us with 18 dialogues in validation data and 17 dialogues for test data. ## results the results in terms of the final output of our best performing incremental bert system in the three testing regimes versus its competitors is shown in disfluency detection on transcripts for repair detection, our system's best f rps score for detecting repair onsets on pre-segmented transcripts at 0.853 beats state-of-the-art incremental systems. this performance degrades using automatic segmentation to 0.802, a state-of-the-art result for this setting. its f rm accuracy of 0.757 on reparandum words on pre-segmented transcripts is only beaten by hp'14/phh'18 model using word and pos input, making it a state-of-the-art strictly incremental deep model. this performance degrades to 0.678 on raw transcripts but is a state-of-the-art result for this setting. in terms of edit term detection, stateof-the-art detection results of 0.960 and 0.944 are achieved on the pre-segmented and unsegmented settings, improving over the existing benchmarks of hp'14 and rh'20. these results suggest we have achieved the aim of a strictly incremental model achieving high final accuracies. ## discussion and conclusion our incremental gpt-2 and bert-driven system performs well at detecting repair disfluencies on pre-segmented and unsegmented transcripts, achieving state-of-the-art results for a strictly incremental repair onset detection. our system is competitive at reparadnum word detection and achieves state-of-the-art results in edit term detection. the results on asr transcripts are also state-of-the-art. the high sequence-final performance comes at the expense of marginally increased jitter in the word-by-word output, but with sensitive and fast repair detection, on average first detecting the repair under a third of a second after the end of the repair onset word. these results suggest it is beginning to enjoy the best of both worlds in leveraging the right-ward context which bert uses for its high performance, while the continuation predictions from the gpt-2 model are good enough to allow good incremental performance before the true right-ward context is available. the linguistic knowledge in the bert model allows it to recognize parallelism in reparandum and repair phases and the absence thereof to increase performance on detecting substitution and delete repairs. this improvement to existing deep disfluency detection models, and, with appropriate use of open-ended language generation techniques with a gpt-2 language model, its good incremental performance, is consistent with a growing body of work @xcite @xcite @xcite , showing good language modelling can lead to good disfluency detection, as they are inherently part of the same process. our system still fails to detect longer repairs compared to an explicit backtracking mechanism like @xcite . while the vanishing gradient problem is partly overcome here, the strictly left-to-right constraint on decoding puts memory limitations on any repair detection system. in future, we will explore efficient ways to navigate this space whilst not filtering out rarer repair forms. the results on asr results show our disfluency detection system is ready for use in a live setting with a good degree of accuracy, and work is currently underway to use it to help detect a variety of different cognitive conditions, including alzheimer's disease, in a live diagnostic system.