Incremental Disfluency Detection for Spoken Learner E nglish (2022)

Abstract: Incremental disfluency detection provides a framework for computing communicative meaning from hesitations, repetitions and false starts commonly found in speech. One application of this area of research is in dialogue-based computer-assisted language learning (CALL), where detecting learners’ production issues word-by-word can facilitate timely and pedagogically driven responses from an automated system. Existing research on disfluency detection in learner speech focuses on disfluency removal for subsequent downstream tasks, processing whole utterances non-incrementally. This paper instead explores the application of laughter as a feature for incremental disfluency detection and shows that when combined with silence, these features reduce the impact of learner errors on model precision as well as lead to an overall improvement of model performance. This work adds to the growing body of research incorporating laughter as a feature for dialogue processing tasks and provides further support for the application of multimodality in dialogue-based CALL systems.

## introduction speech disfluencies such as hesitations, repetitions and false starts are an inherent artefact of spoken language. systematic in their structure, disfluencies comprise of a reparandum phrase, optional interregnum phrase and repair phrase @xcite . i'd like a coffee reparandum + {uh} interregnum tea repair please following the notation scheme described by @xcite , the example above shows the components of a disfluency. the speaker changes their drink order by replacing "coffee" with the intended word "tea". the + represents the 'interruption point', often marked prosodically with features such as silence or reparandum word cutoff. the following, optional interregnum phrase can contain filled pauses such as "uh" like in the example, edit terms such as "i mean" and finally discourse markers such as "you know". detecting such disfluencies is of particular interest in the context of dialogue-based call, where learners interact with an automated system in order to practice conversation in the language that they are learning. in the task-based scenario where a learner is practicing ordering a drink at a café, having a system that can identify and appropriately respond to learners' disfluencies in real-time is highly desirable and not something that is available in existing approaches thanks to the lack of incremental processing @xcite . with the above in mind, this work builds on incremental disfluency detection research and applies it to a language learning setting. the nature of disfluencies in learner speech are explored and learner errors are identified as an area of difficulty in existing approaches. subsequently, the non-lexical features of laughter and silence are suggested as possible solutions to this issue and their impact is tested and compared to a baseline model. findings are reported and considerations for future work in this area are discussed. ## related work disfluency detection is a widely studied area of research, with the most successful approaches leveraging bert transformer models to achieve high accuracy (e.g. @xcite @xcite . these models operate non-incrementally using whole sentences as inputs, often with a view to remove the disfluencies from transcripts all together. this is also the case for research on disfluency detection in learner speech, which has been applied to improve the downstream tasks of grammatical error detection and correction using bi-directional lstms @xcite as well as end-to-end mod-els @xcite . approached as a sequence labelling task, disfluencies are flattened and models are trained to detect the reparandum phrase. this approach is not suited to spoken dialogue systems, however, which benefit from word-by-word processing and the retention of all parts of the disfluency in order to generate meaningful and timely responses @xcite . in a language learning context, such capabilities would not only enable conversational systems to better employ incremental feedback strategies such as prompting but also provide insight into the nature of individual learners' disfluency behaviours. incremental disfluency detection addresses the issues described above and forms a smaller subsection of research. restricted by their left-to-right operability, incremental systems detect disfluency at the point of repair onset and subsequently 'lookback' for the reparandum phrase. to date, there has only been one research paper related to incremental disfluency detection for learner speech, where @xcite reported poor performance when using an incremental dependency parser trained on native data. various approaches have been tested using the switchboard corpus @xcite , however. these are described below. following a noisy channel approach, @xcite implemented a pipeline of random forest classifiers detecting interregna, repair and reparandum phrases separately using input features derived from trigram language models for words and pos tags. simplifying the task to a one model, multi-class sequence labelling problem using deep neural networks, @xcite successfully applied a rnn using only word embeddings and pos tags as input features. this approach was extended further, using lstms for the joint tasks of utterance segmentation @xcite as well as multi-task learning with utterance segmentation, pos tagging and language modelling @xcite . current state-of-the-art performance is achieved by @xcite , who incrementalised a bert-based disfluency detector by using utterance predictions from a gpt-2 language model as inputs to the model. with the exception of word timings @xcite @xcite the incremental approaches outlined above have yet to explore the impact of non-lexical fea-tures on disfluency detection, despite having been successfully integrated into non-incremental settings @xcite . considering the fact that incremental detection begins at repair onset, it seems likely that leveraging paralinguistic information associated with the interruption point will be beneficial to detection. approaches to such integration are explored in this work. ## disfluencies in learner speech on average, disfluencies occur at a higher rate in learner speech compared to native speech @xcite . learner speech disfluency datasets also contain longer reparandum phrases compared to native equivalents @xcite . this is in part thanks to language learners having a lower degree of 'automatisation' in the language they are learning @xcite and is cited by @xcite as the reason why disfluencies in learner speech are more difficult to detect automatically. another artefact of learner speech disfluencies is their co-occurrence with learner errors. the examples below highlight how errors interact with disfluencies in the nict-jle corpus used for this study. the disfluency phrases are labelled and words in bold indicate learner errors. as the examples show, learner errors can occur in the reparandum phrase, the repair phrase, or both. the first example shows an instance where the learner error occurs in the reparandum phrase and is then subsequently repaired to its correct form. the second example shows how this can occur in a nested disfluency, where the inner disfluency instance contains learner errors in both the reparandum and repair phrases, with the outer disfluency instance being without error. the third example shows an instance where the initial reparandum phrase is correct but the subsequent repair phrases both contain errors. the presence of learner errors is often cited as a contributing factor to the difficulty of other nlp tasks for learner language data such as parsing @xcite and pos tagging @xcite . with this in mind, it was hypothesised that learner errors would have a similar negative effect on disfluency detection and so their impact was tested as part of this experimentation. ## silence and laughter incorporating instances of silence is a successful method of increasing model performance in nonincremental disfluency detection research. silence has been encoded explicitly using its presence or absence as an input feature @xcite , implicitly through the inclusion of audio features such as filter banks @xcite and even as a prediction of prosodic cues from text @xcite . research into the nature of silence in learner speech has shown that non-native speakers are more likely to pause mid-clause than native speakers during linguistic processes such as repair @xcite . with this in mind, it seems likely that including silence features will have a positive impact on the model performance and so is tested here. language learners use laughter as a 'trouble management device' during uncertainty (looney and he, 2021), when pre-empting a problematic action @xcite and after making an error @xcite . in an analysis of uk university english proficiency interviews of 23 chinese students, gao (2020) found that laughter co-occurs with disfluencies in three ways: (i), on its own between the reparandum and repair phrase, (ii), alongside indicators of an interruption point such as pauses and word cutoffs, and (iii), simultaneously as laughed speech either during the repair phrase or the whole disfluency. laughter has been shown to improve performance of models for other dialogue processing tasks such as dialogue act classification @xcite but as of yet, has not been applied as a feature to detect disfluencies in learner speech. ## discussion the results from this experimentation give support to the integration of paralinguistic features for incremental disfluency detection in learner speech. the impact of silence and laughter on the detection precision of disfluencies that co-occur with learner errors highlights the value of such features in settings where lexical data is 'non-typical'. this is of particular importance in incremental approaches where detection occurs at repair onset, with a reduced reliance on the syntactic parallelism between reparandum phrase and repair phrase often exploited by non-incremental systems. despite the improvements described above, overall performance gains are small and remain lower than non-incremental approaches. however, there are further approaches to model improvement worth exploring. firstly, following the recent work of @xcite , it would be of interest to test the impact of an incrementalised bert-based detector on learner speech. secondly, using a pos-tagger specifically for learner speech such as that developed by @xcite may help boost performance. it would also be beneficial to investigate the impact of these adaptations on other aspects of learner speech that inform disfluency behaviour, including learners' first language, task type and proficiency level. another limitation of the study is that the nict-jle corpus is a transcription-only dataset with limited features. without audio files available, instances of silence and laughter are derived directly from transcripts. in the same way that asr output deteriorates disfluency detection performance compared to transcribed data @xcite , it is likely that automatic laughter and silence de-tection derived from audio would have a similar effect and may not be as impactful for model improvement. in addition, it would be interesting to investigate the relationship between learner errors and disfluencies by modelling these features jointly. however, in the nict-jle corpus, learner error tags are only available for the test set and so cannot be used as features in training. furthermore, the performance boost shown when combining laughter together with silence provides the motivation to explore additional paralinguistic features in combination, such as gestures and gaze, both of which have been shown to be used in conversation to signal disfluency @xcite . finally, as the nict-jle corpus is a collection of assessor-learner conversations, it is not clear if learners would still enact the same strategies of laughter to indicate disfluencies when practising with a dialogue-based call system. ## future work to the best of our knowledge, there is currently no publicly available resource that addresses the limitations of the nict-jle corpus outlined above. with this in mind, there is a strong case to be made for the development of a multimodal corpus for use in dialogue-based call applications, collected by means of a 'wizard of oz' experiment with language learners and human language tutors. audio, video and transcript files annotated with disfluencies, edit terms, learner errors as well as paralinguistic information would provide ample opportunity for research into both incremental disfluency detection and also other dialogue processing tasks. ## conclusion in conclusion, this work tested the impact of laughter and silence as features for incremental disfluency detection of learner speech. when combined, these features show an overall improvement in model performance, increasing precision for disfluencies that co-occur with learner errors. to date, this is the first work to use laughter as a feature for disfluency detection in a language learning setting, with the resulting model significantly outperforming previous incremental approaches for learner speech. these findings act as a starting point for the further integration of paralinguistic features for incremental disfluency detection and help make the case for the development of a multimodal corpora for dialogue-based call applications.