How You Say It Matters: Measuring the Impact of Verbal Disfluency Tags on Automated Dementia Detection (2022)

Abstract: Automatic speech recognition (ASR) systems usually incorporate postprocessing mechanisms to remove disfluencies, facilitating the generation of clear, fluent transcripts that are conducive to many downstream NLP tasks. However, verbal disfluencies have proved to be predictive of dementia status, although little is known about how various types of verbal disfluencies, nor automatically detected disfluencies, affect predictive performance. We experiment with an off-the-shelf disfluency annotator to tag disfluencies in speech transcripts for a well-known cognitive health assessment task. We evaluate the performance of this model on detecting repetitions and corrections or retracing, and measure the influence of gold annotated versus automatically detected verbal disfluencies on dementia detection through a series of experiments. We find that removing both gold and automatically-detected disfluencies negatively impacts dementia detection performance, degrading classification accuracy by 5.6% and 3% respectively

## introduction as populations grow older worldwide, the number of people with alzheimer's disease (ad) and related dementia is also on the rise (alzheimer's association, 2018). significant changes to speech and language use caused by dementia occur early in disease progression @xcite . interesting case studies have demonstrated how diachronic analysis of patients' language use may reveal signs of dementia, using writing samples from british novelists iris murdoch, who ultimately perished with alzheimer's, and agatha christie, who was suspected of it @xcite . numerous studies have also sought to automatically detect early signs of the disease and model its progression using speech and writing samples @xcite @xcite @xcite @xcite @xcite . although some studies have pointed to disfluency patterns as an important predictor of ad status @xcite , research in this area has been limited by several factors. disfluency detection is a challenging and resource-intensive task in itself @xcite @xcite , and may lie out of scope for many interdisciplinary researchers already straddling boundaries between nlp and clinical practice @xcite . rich manual disfluency annotations are present in some datasets common in automated dementia detection @xcite , but off-the-shelf asr systems do not typically transcribe disfluencies. moreover, inconsistencies between automatically generated and gold standard transcripts may pose significant challenges for modeling dementia in real-world applications @xcite , for which asr will be a necessary component of any speech-based pipeline. we address these limitations, by investigating the impacts of automatically derived disfluencies on modeling cognitive decline. our key contributions are as follows: this analysis 1 not only paves the way for the discovery of approaches to automated dementia detection that are more suitable for realistic scenarios, but also enhances our understanding of the individual contributions of different disfluency types to this task. we report on related studies and provide relevant background for automatic disfluency detection in §2. we describe our datasets and task setup in §3, and detail our methods in §4. we report the results of our experiments in §5, and further analyze our findings in §6 before concluding in §7. ## data and task setup we used the adress challenge corpus for our experiments @xcite . the adress challenge corpus, developed as part of a shared task for interspeech 2020, is a benchmark dataset of spontaneous speech in the domain of ad classification and mmse score prediction. it has been acoustically preprocessed, and is balanced in terms of age and gender. the data consists of audio recordings and manual transcriptions of spoken picture descriptions elicited from participants through the cookie theft task from the boston diagnostic aphasia exam @xcite . the corpus is a subset of the pitt corpus, foot_0 which is itself a subset of the dementiabank dataset @xcite . in the cookie theft task, an investigator and a participant (in this case, an older adult) carry on a conversation in which the investigator asks the participant to describe what is depicted in an eventful image containing, among other subjects, a boy stealing a cookie from a cookie jar. 3 there is no specific time limit for the conversation, allowing participants to talk as long as they want. in the pitt corpus and by extension the adress challenge corpus, these conversations were recorded and manually transcribed using the chat transcription protocol @xcite . participants were labelled as hc (healthy control with no cognitive decline) or ad (declined cognitively) based on their prior diagnostic test results. ## discussion from our corpus analyses, we find that members of the ad group exhibit more verbal disfluency ( ## conclusion verbal disfluencies are an important indicator of ad, and current asr systems fail to capture and label word-and phrase-level disfluencies adequately. doing so is necessary to generate useful transcripts with minimal human intervention, such that they can be leveraged for successful ad detection. our future work will focus on training an end-to-end asr system on disfluent speech so that it can generate richer disfluency annotated transcripts, which will pave the way for building end-to-end speechbased dementia detection systems.