Towards Hierarchical Spoken Language Disfluency Modeling (2024)

Abstract: Speech dysfluency modeling is the bottleneck for both speech therapy and language learning. However, there is no AI solution to systematically tackle this problem. We first propose to define the concept of dysfluent speech and dysfluent speech modeling. We then present Hierarchical Unconstrained Dysfluency Modeling (H-UDM) approach that addresses both dysfluency transcription and detection to eliminate the need for extensive manual annotation. Furthermore, we introduce a simulated dysfluent dataset called VCTK++ to enhance the capabilities of H-UDM in phonetic transcription. Our experimental results demonstrate the effectiveness and robustness of our proposed methods in both transcription and detection tasks.

## introduction spoken language disfluency 1 modeling is the core technology in speech therapy and language learning. according to @xcite , an estimated 17.9 million adults and 1.4 percent of children in the u.s. suffer from chronic communication and speech disorders. currently, hospitals have to invest substantial resources in hiring speech and language pathologists @xcite to manually analyze and provide feedback. more importantly, the cost is not affordable for low-income families. kids' speech disorders also have a significant connection to the language learning market. according to a report by @xcite , the english language learning market will reach an estimated value of 54.8 billion by 2025. unfortunately, there is not an ai tool that can effectively automate this problem. in current research community, there is not a unified definition for disfluent speech, as mentioned in @xcite . as such, we solidify the definition of disfluent speech as any form of speech characterized by abnormal patterns such as repetition, replacement, and irregular pauses, as summarized in @xcite . within the domain 1 disfluency is interchangable with dysfluency of disfluent speech modeling, research efforts are conducted both on the speech side and the language side. whenever disfluent speech transcription is given (such as human transcription in figure 1 ), the problem can be tackled by llms @xcite . however, such transcription is not available and current best asr systems such as @xcite tend to recognize them as perfect speech. thus, we argue that the bottleneck lies in the speech side rather than in language. unfortunately, there is also no established definition for the problem of speech disfluency modeling. we formally define that speech disfluency modeling is to detect all types of disfluencies at both the word and phoneme levels while also providing a time-stamp for each type of disfluency. in other words, disfluency modeling should be hierarchical and time-accurate. previous research has mainly focused on a small aspect of this problem. researchers started by focusing on spotting stuttering using end-to-end methods. they manually tagged each utterance and developed the classification model at the utterance level @xcite @xcite . later on, things got detailed with frame-level stutter detection @xcite . however, end-toend methods have their limitations. first, stuttering is just one aspect of disfluency. current end-to-end models struggle to handle other forms of disfluency effectively. second, manually labeling data for these methods is a lot of work and not practical for larger-scale projects. lastly, disfluency modeling depends on the specific text being spoken, a factor that has been overlooked in previous research, as pointed out in @xcite . it is typically intuitive to consider speech transcription that offers disfluency-specific representations. for a long time, the mainstream of researchers in speech transcription has been focused on word-level automatic speech recognition (asr), unconstrained recursive forced aligner (urfa) whisper text refresher reference text: you wish to know all about my grandfather "real"(human) transcription: you wish [] know [pause ao l pause ah b pause] all about my grandfather hierarchical unconstrained dysfluency modeling(h-udm) you wish to know all about my grandfather you wish [] know all about my grandfather large discrepancy 1 2 3 forced alignment ref text smoothed re-segmentation unconstrained forced aligner (ufa) 2d-aignment search unconstrained recursive forced aligner (urfa) zero-order urfa transcription module detection module silence 2d-alignment 2. irregular pause: [4.60s-5.50s, 6.06s-7.30s, 7.64s-8.80s] 1. phonetic error: [4.48s-4.52s] 3. repetition: [5.60s-9.10s] word missing: "to" word repetition: n/a word insertion: n/a …… word dysfluency phonetic dysfluency 1st-order urfa 2nd-order urfa 3rd-order urfa …… t e m p la t e m a t c h in g which has been further scaled. however, the most advanced word transcription models currently available @xcite @xcite @xcite can only transcribe certain obvious word-level disfluency patterns, such as word repetition or replacement. however, the majority of disfluencies occur at the phoneme-level or subwordlevel, making them challenging for any asr system to explicitly detect. @xcite introduced a neural forced aligner that incorporates time accuracy and sensitivity to silence. this aligner employs a weighted finite-state transducer (wfst) to capture disfluency patterns like repetition. however, it fails on openset disfluency modeling @xcite . the unconstrained disfluency model (udm) introduced in @xcite was devised to address the aforementioned challenges comprehensively. udm seamlessly integrates both transcription and detection modules within a unified framework. within the udm framework, non-monotonic alignments are acquired through dynamic alignment search, forming the foundation for subsequent template matching algorithms aimed at detecting various disfluency types. specifically, distinct templates are tailored for each disfluency category, encompassing replacements, insertions, deletions, blocks, and repetitions. additionally, vctk++ dataset was introduced to further enhance model performance. in the present study, we extend the capabilities of udm by incorporating a monotonicity constraint. while non-monotonic alignment is essential for effective disfluency modeling, our experiments demonstrate that the integration of a simple connectionist temporal classification (ctc) module alongside a phoneme classifier can enhance non-monotonicity. furthermore, we introduce the unconstrained recursive forced aligner (urfa), which employs an iterative process to generate both phoneme alignments (1d) and 2d alignments with weak text supervision. this recursive modeling significantly enhances detection robustness. our proposed method, termed hierarchical unconstrained disfluency modeling (h-udm), attains state-ofthe-art performance in real aphasia speech disfluency detection. ## transcription module our transcription module consists of two core parts: transcription module1: unconstrained recursive forced aligner (urfa) 2d-alignment search smoothed re-segmentation 1st-order urfa silence you wish to know all about my grandfather silence ufa ufa ufa …… 2d-alignment search 2d-alignment search 2d-alignment search smoothed re-segmentation …… zero-order urfa 3rd-order urfa …… concatenation detection detection forced alignment ref text 2nd-order urfa boundary update ufa …… …… …… in the first iteration (zero-order), the entire utterance is taken and 2d alignment is generated. starting at 2nd iteration (1st-order), the disfluent speech is segmented at word level and each segment is processed separately and then combined to generate the final 2d alignment for detection. ## detection module the reason we adopt a separate design for the detection and transcription modules is that an end-toend modeling approach for the detection system is not reliable. the transcription module provides us with disfluency-aware representations to optimize the detection module. here, we can still design learning-based methods @xcite @xcite to predict the detection results; however, we don't have human labels for disfluencies, which might be considered for future work. instead, we have developed a smart label-free system that simply employs the template matching algorithm for each type of disfluency. template matching is efficient and reliable, eliminating the need for human annotation. we have designed disfluency templates for both word and phoneme levels. these disfluencies include phonetic errors (missing, deletion, replacement), repetition, and irregular pause. our methods also cover word-level disfluencies, including missing, insertion, replacement, and repetition. the following section details them. ## limitations we propose a hierarchical unconstrained dysfluency modeling (h-udm), which is an extension of udm @xcite . h-udm introduces ctc monotonicity, and the incorporation of recursive modeling significantly enhances both transcription and disfluency detection results by a substantial margin. however, there are still several limitations to consider. first, the results on disordered speech are not as satisfactory. this suggests that the inference-only algorithm, the template matching algorithm, may not be sufficient for advanced disfluency modeling. it remains essential to develop end-to-end methods to address this challenge, which, however, presents its own set of difficulties. second, the current closed-set disfluency classification only includes five types of disfluencies: replacement, insertion, repetition, block, and deletion. however, in an open-domain disfluency modeling system, there are many other complex disfluency patterns, such as syllable swapping and false starts. designing specific templates for each type of disfluency is impractical. third, phoneme units may not be the optimal choice for modeling disfluency. for instance, there may not be significant acoustic differences between "ah" and "aa," yet h-udm treats them as two distinct phonemes. although this is partially alleviated by smoothed segmentation, the improvement is limited. therefore, it is worth exploring alternative speech units, such as articulatory units @xcite @xcite , to enhance alignment modeling. ## acknowledgement thanks for support from uc noyce initiative, society of hellman fellows, nih/nidcd and the schwab innovation fund. figure 8: segmentation-(aphasia sample: usually several buttons missing.) figure 9: segmentation-(my stutter sample: please call stella.)