Uncovering the Potential for a Weakly Supervised End-to-End Model in Recognising Speech from Patient with Post-Stroke Aphasia (2023)

Abstract: Post-stroke speech and language deficits (aphasia) significantly impact patients’ quality of life. Many with mild symptoms remain undiagnosed, and the majority do not receive the intensive doses of therapy recommended, due to healthcare costs and/or inadequate services. Automatic Speech Recognition (ASR) may help overcome these difficulties by improving diagnostic rates and providing feedback during tailored therapy. However, its performance is often unsatisfactory due to the high variability in speech errors and scarcity of training datasets. This study assessed the performance of Whisper, a recently released end-to-end model, in patients with post-stroke aphasia (PWA). We tuned its hyperparameters to achieve the lowest word error rate (WER) on aphasic speech. WER was significantly higher in PWA compared to age-matched controls (10.3% vs 38.5%, p<0.001 ). We demonstrated that worse WER was related to the more severe aphasia as measured by expressive (overt naming, and spontaneous speech production) and receptive (written and spoken comprehension) language assessments. Stroke lesion size did not affect the performance of Whisper. Linear mixed models accounting for demographic factors, therapy duration, and time since stroke, confirmed worse Whisper performance with left hemispheric frontal lesions.We discuss the implications of these findings for how future ASR can be improved in PWA.

## introduction aphasia is a language impairment that causes difficulties in speaking, understanding and/or writing coherent and meaningful sentences. this deficit negatively impacts numerous daily activities, such as working, shopping or participating in community and leisure experiences. as a consequence, patients with aphasia report high levels of depression, passiveness, social exclusion and a general decline in their quality of life @xcite . overall, there are at least 2 000 000 people in the usa (national aphasia association) and more than 350 000 people in the uk with aphasia (stroke association). roughly 45% of aphasic disorders arises following a stroke @xcite . stroke cases, mortality and morbidity have increased substantially over the last two decades, with 70% increase in incident strokes, 43% deaths from stroke, and 143% dalys @xcite . consequently, the incidence of aphasia has also increased. importantly, the presence of aphasia per se worsens the overall stroke outcomes @xcite . therefore, due to the psycho-social burden and the current increase in stroke cases, early diagnosis and treatment of aphasia need to be addressed. the mainstay treatment of aphasia is speech and language therapy; it entails practices with language exercises for improving language ability, as well as adjusting to new ways of communicating @xcite . according to the results of different meta-analyses, higher intensity speech therapy treatment is strongly associated with greater treatment efficacy @xcite @xcite . providing ongoing efficient treatment, however, can be challenging due to limited resources, which can make face-to-face speech therapy costly and difficult to achieve for every patient need @xcite . the situation became worse especially after the covid-19 pandemic crisis, that led to the suspension or the slowdown of non-urgent care, including speech and language therapies @xcite . a solution for these issues might be the use of speech recognition models, able to remotely and automatically transcribe long pieces of conversation to easily analyse patients language profiles and to give tailored treatments. nevertheless, even though automatic speech recognition (asr) tools have been already explored in research, until now these have been slow to catch up with the performance obtained in healthy speech @xcite @xcite . indeed, the models trained on healthy data struggle to achieve high accuracy in metrics like word error rate (wer) or phoneme error rate (per), mostly due to the features of aphasic speech. speech from pwa is largely thought to have semantic (meaning) and phonological (speech sound) errors, as well as dysfluencies, each with independent recovery trajectories @xcite . furthermore, aphasic speech has characteristics that might include: slow and hesitant elocution with episodes of agrammatism (e.g. absence or improper use of function words and verbs -damico et al. 2010), word-finding problems that affect mostly nouns and picturable action words, frequent stammer, as well as an overall flow of speech that is often fragmented, choppy, unintelligible and/or awkwardly articulated @xcite . these aspects can be influenced also by motor control problems like apraxia and dysarthria, frequently present in aphasia, which may also produce articulation distortion and aberrant prosody @xcite . hence, the challenges that these models need to address include the high variability of speech errors, both between and within aphasic individuals, as well as the lack of satisfactory training datasets. we tested the performance of a state-of-art sequence-to-sequence asr transformer model that, to our knowledge, has not been used yet on clinical data. this model, released by open ai @xcite , is named whisper 1 and it is known for its superior performance in healthy speech when compared with other notable commercial and opensource asr systems. the feasibility of this model in clinical practice is supported by the low wer in healthy speakers, and the powerful large multilingual weakly supervised dataset on which it was trained. moreover, the ability to run whisper locally, will help to preserve the privacy of patients' sensitive data and allow testing in compliance with local and continental regulations. for the purpose of this study, the whisper testing is done on a novel database of speech of pwa that we have created. we fine-tuned whisper parameters relevant for aphasic speech, detailed in section 3.2. this led us to retrieve the best model according to the low1 the model name comes from the acronym of wspsr standing for web-scale supervised pretraining for speech recognition est wer to test on speech audio. we then compared the patients' wer to an aged-matched control group that performed the same speech production task. after correlation analyses, we created linear mixed-effects models and observed interesting and significant relations with the average performance of the asr (see section 4). according to these results, our analysis offers useful insights to consider for our next steps, from which other researchers can also benefit. we expect that our study will advance the work of asr for pwa, enriching and inspiring the research of the natural language processing community applied in the healthcare framework. ## related work since the introduction of asr technology in clinical studies, algorithms have had to deal with several challenges. the variability and complexity of disordered speech, sometimes unintelligible, has led researchers to move forward with the creation of novel asr trained with pathological speech data. nevertheless, an additional difficulty they have to face is the scarcity of datasets of such disordered speech, limiting the accuracy and/or the generalisability of the results. an example of this is the work of @xcite , which extracted language features from their corpus for distinguishing different frontotemporal lobar degeneration, one of which includes progressive nonfluent aphasia. although the study demonstrated encouraging outcomes, it was conducted on a comparatively limited dataset, and no examination was performed regarding the reliability of the features extracted using asr. similarly, @xcite attempted to differentiate and diagnose primary progressive aphasia (ppa) and two of its sub-types, semantic dementia (sd) and progressive non-fluent aphasia (pnfa) extracting 58 lexical and syntactic features. using a reduced dataset, an optimized support vector machine (svm) and random forests (rf) classifiers, @xcite tried to face the problem of the dataset with data augmentation on a recognition model for patients with dysarthria. they reached an overall wer of 27.8% on the uaspeech test set, underlining that the lowest published wer on the subset of speakers with "very low" unintelligibly was of 57.3%. differently, kohlschein and colleagues (2017) used the speech elicited in the aachen aphasia test assessment as a database to train their algorithm. they built a model that automatically analysed pathological speech to identify patients' aphasia type and severity based solely on acoustic features. aphasiabank, a large database open to members, was used by @xcite to successfully detect medically-relevant quantitative measures to predict aphasia with wer (word error rate) of 39% in spontaneous aphasic speech. their previous work, with a wer of 45% @xcite , established the first asr baseline on aphasiabank, showing that this dataset can guide the understanding of aphasic speech recognition. in @xcite , the authors used an acoustic modelling architecture of multi-task dblstm-rnn (double bidirectional long short-term memory recurrent neural network) with four hidden blstm with 2 diverse language models for decoding. the authors investigated features based on speech duration, the quality of pronunciation, phone edit distance, and dynamic time warping on phoneme posteriorgrams. on the other hand, @xcite , even though using a similar pipeline, chose to investigate lexical diversity and complexity, posteriorgram-based dynamic time warping, pairwise variability error, dysfluency and information density in aphasic speech. lastly, like the work of @xcite , a cantonese version of aphasiabank has been implemented by @xcite , together with the cusent and canpev cantonese corpora. in this case, as an evaluation metric they used a syllable error rate (ser) with the aphasiabank and a multilayer time delay neural network (mt-tdnn) with a bidirectional long short-term memory (blstm) model structure. this obtained 18.5% of wer for unimpaired speech and 42.4% for impaired speech. an alternative strategy is to employ asr that already exists as per study by mahmoud and colleagues (2023), where the authors customised existing asr for a specific research goal, selecting microsoft azure speech-to-text api or google speech-to-text api. in this study we are adopting a similar approach: given the impressive performance of whisper on healthy speech, largely due to its training dataset being several orders of magnitude larger than preceding asr, we expect whisper accuracy to be similar to aforementioned models trained on aphasic data. we used as input to whisper the data of 46 participants, divided into an aged-matched controls group (@xmath0; @xmath1. for patients, audio speech was collected across multiple time-points since their stroke, resulting in a total of 38 audio files. the speech is recorded during the picture description task from the cat assessment @xcite . the audio was transcribed verbatim by a speech therapist and 3 trained postgraduate students, with excellent inter-rater reliability (73% overall wordlevel match). the text is in chat format (codes for the human analysis of transcripts; macwhinney 2014), managed and analyzed through the clan software. using clan, the following measures were generated: grammatical complexity (mean length of utterance in morphemes), productivity (number of total words), lexical diversity (number of different words), fluency (words per minute) and flawed syntax (incorrect utterances that do not have at least one verb, copula, modal, or participle). all these measures are included in the sample characteristics in ## results through the grid search optimisation, we generated a total of 48 models, obtained by the combinations of the three aforementioned parameters. the model that performed best, according to the lowest wer, was the one that used the medium model, with compression_ratio_threshold at 2.0, and logprob_threshold at -1.5. the wer differed significantly between controls and patients (@xmath2, @xmath3.001, fig. 1 ), patients had an almost four-fold increase in wer than the control group (38.5% vs 10.3%). considering the correlation analysis, cat scores and wer associations were all found to be significant and they are shown in figure 2 . all the three scores of cat showed a negative relation with the outcome, reflecting in general a worse precision of the asr in the case of patients with more severe aphasia. as far as the stroke lesion volume is concerned, no significant correlation was found. since we wanted to adjust results for potential confounders and find significant and meaningful relations, we modelled the data with mixed-effect models and reported the outcomes in table ??. in total, four main clusters of models were run to evaluate the effects of lesioned hemisphere, lesion presence considered singularly, cat scores, and lesion volume on the abilities of the asr to transcribe correctly the speech. all the models were adjusted for socio-demographic (age, gender and years of education) and aphasia-related information (time of test since stroke and hours of speech therapy). comparing patients with lesions in left and right hemispheres, the asr performed worse in terms of wer in left temporal and frontal lobes, as well as in the left parietal lobe, although this last association was not statistically significant. testing individual lobes confirmed that asr performance is worse in patients with left frontal lobe lesions, linking it to the localization of expressive language. moreover, even considering these models, the lesion volume features did not show any significant result. finally, cat derived total picture description score, naming score and comprehension score all had negative relations with the wer, representing higher errors when patients performed poorly in the tests. ## discussion the evaluation of the asr using the wer metric allowed us to understand how well whisper, a model trained on a very large healthy speech dataset, performs on pwa speech. we were able to optimise the performance of whisper based on three hyperparameters, observing similar outcomes in terms of wer when comparing the performance of fine-tuned whisper model with the performances of previously described asr systems tailored for pwa. using two measures of overt speech production (cat naming and cat total picture description score for spontaneous speech production) and a measure of speech comprehension (cat comprehension score), we were able to show that asr performances is related to the severity of aphasia. these results were confirmed by the mixed-effect models when adjusting for confounding factors such as demographics, time since stroke or duration of therapy. our findings are in keeping with the study by @xcite that reported 55.5% wer in severe and 22% in mild cases of aphasia. furthermore, we showed for the first time that stroke lesion location is related to the performance of the asr. speech from patients with left lateralised lesions, and more specifically in the left frontal lobe, was the hardest to recognise using whisper. this result is consistent with the known localisation of spoken language processing in the brain. specifically, frontal lobes, together with other parts of the language network, are thought to be primarily implicated in higher-order language functions, such as sentence comprehension, production, speech planning and overt speech production @xcite . temporal lobes are essential for language processing and retrieval of semantic information during overt naming @xcite . future studies can use information about stroke lesion or brain anatomy to improve asr training and performance in pwa. qualitatively, we noted in some cases whisper was capable of transcribing filler words (such as "hum", "umm"), frequently observed in pwa. despite this, the wer occasionally increased as a result of the frequent usage of fillers. false starts (e.g. 'the k-kit-umm... the kitty') were rarely detected and transcribed correctly. there were cases when some words were uttered with low speech volume and were not detected at all, as well as unintelligible words that were skipped altogether by whisper. these qualitative observations need to be validated with quantitative analysis on larger aphasia-specific datasets to identify pwa speech features that contribute to the worse performance of asr in pwa. the 'confidence' of the asr in detecting these aspects can accordingly be reduced, and more specific asr training can be performed on speech encompassing these specific features. ## conclusion and future work this study evaluated the performances of the whisper end-to-end asr model on speech derived from patients with post-stroke aphasia. the results highlight the importance of taking lesion location and stroke severity into account when developing speech therapy diagnostics or interventions for pwa using asr models. our findings require verification in larger speech databases derived from patients with post-stroke aphasia and their generalisability needs to be assessed in cases of aphasia resulting from other conditions, such as neurodegenerative dementias, which may have different characteristics. despite fine-tuning the in-built whisper parameters to optimise the model performance in this clinical population, we demonstrated that even though whisper has a competitive performance compared to existing aphasia-specific asr, it still lacks sufficient clinical diagnostics accuracy. furthermore, additional asr metrics such as the confidence of the asr transcription or the phoneme error rate could be adopted in future research. a further limitation of this work is the small speech database used in this paper. we are actively building a detailed annotated large speech and language database from hundreds of patients with post-stroke aphasia, with the aim of training and developing asr for pathological speech. we expect that such work will promote greater confidence in the use of ai and specifically nlp for healthcare intervention.