Out of the Box, into the Clinic? Evaluating State-of-the-Art ASR for Clinical Applications for Older Adults (2025)

Abstract: Voice-controlled interfaces can support older adults in clinical contexts – with chatbots being a prime example – but reliable Automatic Speech Recognition (ASR) for underrepresented groups remains a bottleneck. This study evaluates state-of-the-art ASR models on language use of older Dutch adults, who interacted with the Welzijn.AI chatbot designed for geriatric contexts. We benchmark generic multilingual ASR models, and models fine-tuned for Dutch spoken by older adults, while also considering processing speed. Our results show that generic multilingual models outperform fine-tuned models, which suggests recent ASR models can generalise well out of the box to real-world datasets. Moreover, our results indicate that truncating generic models is helpful in balancing the accuracy-speed trade-off. Nonetheless, we also find inputs which cause a high word error rate and place them in context.

## introduction although there is a surge of interest in ai-driven applications like chatbots in the health domain @xcite , tailoring them to groups underrepresented in ai research remains a challenge. older populations are one example: because they often have different needs and preferences when interacting with ai @xcite , their involvement in system development is key for building clinically relevant systems in geriatrics, the field in healthcare concerned with the health of older adults. this population is increasing in size globally @xcite , while personnel shortages in healthcare become pressing @xcite ); yet addressing these challenges with ai warrants systems that align well with older adults. voice control is a key element in chatbots in geriatrics, as older individuals may struggle with small fonts, icons, and typing text in standard interfaces @xcite . yet, implementing voice control is not obvious, as the performance of automatic speech recognition (asr) systems depends on the representation of older adults in training data, and on their articulation, speech volume and technological literacy @xcite . moreover, evaluation of state-of-the-art asr systems on realistic data of older adults is lagging behind. in this short paper, we evaluate recent asr models on older adults' language use in interaction with welzijn.ai. this is a new digital platform for older users, that currently features a prototype chatbot to converse about clinically relevant topics like quality of life and frailty. audio data of 10 older dutch adults interacting with welzijn.ai were transcribed using generic multilingual asr models (whisper and voxtral), and models fine-tuned specifically for dutch or older dutch populations (whisper and wav2vec2). we find that i) generic multilingual models outperform fine-tuned models, and also that ii) truncating larger generic models helps striking a good balance between accuracy and speed. these findings also hold for a similar subset of the mozilla common voice dataset @xcite we use as benchmark. ## background though work on evaluating asr models for older adults exists, extrapolating findings to realistic contexts is hard. performance of asr models sometimes remains implicit in downstream use, for example through evaluation of overlap of linguistic features extracted from model vs. human-generated transcripts @xcite , or by using only part of asr models (like the audio encoder) in predicting cognitive impairment in older persons @xcite . other work tailors asr systems to older adults by drawing on additional databases of individual users @xcite , which is not always feasible in a clinical figure 1 : interface of welzijn.ai with an example conversation. users press the purple button to activate the asr functionality and start responding, after which their speech is transcribed and rendered on the screen. chatbot responses are read out with a text-to-speech model. the 'scores' button shows information extracted on quality of life and frailty, 'settings' allows choosing different asr models, and 'appearance' returns the user to the conversation on display. we focus in this paper on the conversation resulting from interaction with this prototype. context due to privacy concerns. however, @xcite also show that fine-tuning generic multilingual models with speech from older adults increases asr performance. the work by @xcite explores this further, by finetuning whisper as recent asr model @xcite on the dutch jasmin-cgn dataset. this dataset includes language use of older adults in human-machine interaction settings (cucchiarini and van hamme, 2012), which is potentially relevant to welzijn.ai. ## materials and methods ten older dutch adults (≥65 years) from a volunteer panel of the outpatient clinic for geriatrics at the leiden university medical center (lumc) were included. the institutional review board of the lumc approved the study and all participants provided informed consent. participants were instructed not to share personal health information with welzijn.ai, but rather to impersonate a peer. participants interacted individually with the chatbot, with one experimenter standing by. an impression of welzijn.ai is given in figure 1 . here we focus on the system's ability to support conversations; we refer for architectural details to @xcite . the chatbot was driven by the meta-llama/llama-3.3-70b-instruct foot_0 model @xcite , prompted to structure the conversation around the eq-5d and clinical frailty scale @xcite . these are validated geriatric instruments to assess quality of life and frailty, by retrieving information about mobility, mental wellbeing, physical independence, and so on. these instruments can be presented via surveys, in conversations with clinicians, or in our case, by a chatbot (figure 1 ). the chatbot was used on a laptop. interactions took 5-10 minutes and were recorded with a handheld device. the default asr model in welzijn.ai was openai/whisper-large-v3, which in early testing we found to work best. recorded user speech was after the interactions separated from chatbot responses and segmented using pydub foot_1 and pyannote @xcite . we obtained 199 segments with an average length of 3.4 seconds, due to the turn-taking nature of the conversation. our sample totalled 11 min. and 15 sec., so is small, but still valuable as data from clinical contexts is challenging to obtain. besides chatbot data, we also drew 200 random samples from the common voice dataset @xcite of dutch older individuals (≥ 60 years), which totals 17 min., and 37 sec. as this data concerns written text read out loud, it should intuitively be an easier benchmark for asr models. for obtaining gold standard (i.e. human) reference transcriptions, segments were transcribed with the default openai/whisper-large-v3 model, and subsequently corrected by the first author. since the word error rate (wer) as standard metric in the asr field is sensitive to fillers, capitalisation, and punctuation, and since choosing the 'right' reference depends on the use case, we created three types of human and model transcriptions (via rule-based postprocessing) as visible in our motivation for this set of asr models is that they are all small (compared to voxtral's 24b variant for example) and open weights, hence suitable for local/private downstream applications. we take processing time into account since a good balance between accuracy and speed is key in many applications, so include models of different sizes. our code for the asr pipeline is available. 3 all experiments were carried out on a macbook pro m1 16gb using the hugging face ecosystem for asr and pytorch's mps gpu acceleration backend. due to privacy restrictions we cannot share recordings nor transcripts of the interactions. ## results wer and processing time per model are given in ## discussion and conclusion we evaluated state-of-the-art asr models on transcribing language use of older adults interacting with the welzijn.ai chatbot, which was designed for geriatrics. we included various generic multilingual models as well as models fine-tuned on language use of older dutch adults and general dutch. we found that on both welzijn.ai and common voice data, generic multilingual models perform better than fine-tuned models, with openai/whisper-large-v3 as best model achieving wers of .06 and .12 for normalised and orthographic transcriptions of realistic welzijn.ai conversations. interestingly, its truncated variant openai/whisper-large-v3-turbo struck the best balance between accuracy and processing speed, the latter being crucial in chatbot systems used in real-time. this is useful from a systems development perspective, since truncated models may perform well out of the box, without the need for training smaller architectures from scratch, or for additional data for group-or task-specific finetuning. future work should further support this claim by using larger samples across settings. to put our results in perspective, other evaluations of state-of-the-art asr models on the entire dutch subset of the common voice dataset reported wers of .06 by mistralai/voxtral-mini-3b-2507 (liu et al., 2025), and .04 by openai/whisper-v3-large, assuming the strictest evaluation scenario (orthographic). 4 we obtained similar results for our subset of common voice spoken by dutch adults: .04 for the same whisper and .05 for the same voxtral model. hence, it seems that for generic multilingual models, changing the target population does not imply large performance degradation if the task is straightforward (reading text out loud). still, our welzijn.ai data is conversational, hence results are harder to put in perspective. though the wer on orthographic transcripts for welzijn.ai data triples for the same voxtral and whisper models (.17 and .12 respectively) , given the different nature of read speech and conversational language, this is not a dramatic loss of performance. recent work has reported wers for a variety of datasets transcribed with our best performing model openai/whisper-v3-large, as large as .32 for english speech recordings in home environments (berst), and .23 for english meeting recordings (ami) @xcite . we also attempted to categorize wer error types. we saw that mishearings were as frequent as hallucinations. hallucinations, however, are potentially more problematic for systems like welzijn.ai, as for a user who is unable to make sense of the resulting transcription, trust will erode faster compared to a mishearing, which still has some semantic or phonetic similarity. strategies to improve wer and mitigate hallucinations, could include more independent language modelling components that take specific contexts into account. when the audio input for the decoder is noisy, its language prior generates a transcript based on its own distribution instead of the input, which may well be out-of-context. mitigation could involve generating candidate predictions and evaluating their likelihoods from the perspective of a domain-specific language model, or combining the decoder's predicted token probabilities directly with the prediction of a context-specific language model (see also @xcite . hence, some exciting work remains for making real-world impact with recent asr models.