Speech-Controlled Smart Speaker for Accurate, Real-Time Health and Care Record Management (2025)

Abstract: To help alleviate the pressures felt by care workers, we have begun new research into improving the efficiency of care plan management by advancing recent developments in automatic speech recognition. Our novel approach adapts off-the-shelf tools in a purpose-built application for the speech domain, addressing challenges of accent adaption, real-time processing and speech hallucinations. We augment the speech-recognition scope of Open AI’s Whisper model through fine-tuning, reducing word error rates (WERs) from 16.8 to 1.0 on a range of British dialects. Addressing the speech-hallucination side effect of adapting to real-time recognition by enforcing a signal-to-noise ratio threshold and audio stream checks, we achieve a WER of 5.1, compared to 14.9 with Whisper’s original model. These ongoing research efforts tackle challenges that are necessary to build the speech-control basis for a custom smart speaker system that is both accurate and timely.

## introduction health and social care is one of the last major industries to undergo the digital transformation to improve management of information and connectivity @xcite . reasons include challenges relating to data privacy, tech-literacy and scalability in a highly heterogenuous domain @xcite . transformation towards healthcare 4.0 is helped by integrating new artificial intelligence technologies into purpose-built smart devices @xcite . @xcite find that, even with the implementation of real-time electronic record management, healthcare administrators spend a quarter of their time on documentation and, due to typing distractions, information is missed. combined with the job-demanding stresses that care workers experience @xcite it is clear that there is a need for simplified health care record management to help reduce the burden. this would further benefit those cared for as care resources become more optimised. one way to achieve a quicker, more efficient approach to care record management that is both complete and accurate is through automatic speech recognition (asr, @xcite @xcite . in this paper we focus on recognition of spoken english in the uk. however, typical off-theshelf asr models are often trained primarily on american-accented datasets @xcite and health and care in the uk is a diverse industry. this includes variations in dialects across the british isles @xcite , as well as foreign accents from care workers who originate from places such as eastern europe, nigeria, and south asia, amongst others 1 . commercial smart speakers, such as amazon's alexa, showcase the potential of real-time asr in a general home assistant setting (hoy, 2018), and have been used in previous studies to improve well-being in social care @xcite . however, to the best of our knowledge there is currently no device whose primary function is a smart administrative assistant for health and care workers. hence, we have set out to develop a custom-built speaker, starting with new research into the fundamental asr basis. this paper introduces a novel approach and makes the following key contributions: ## accent adaption despite whisper's extensive training, we find that it struggles to generalise to a broad variety of british as well as other foreign accents found in the care sector. @xcite find a similar bias towards north american over other british accents. we start by adapting whisper to better recognise the variations in six different british accents: 'southern', 'northern', 'midlands', 'scottish', 'welsh', 'irish' from the openslr foot_3 dataset of ∼30 hours of spoken english @xcite . with this dataset, we fine-tune whisper's medium.en model, which balances speed with accuracy, and is the largest model that we can enforce with english-only recognition; the larger models would occasionally incorrectly recognise speech as a different language and attempt to translate. furthermore, the large model requires twice the vram but offers diminishing returns in performance @xcite and we do not require the additional feature of multi-language asr. our fine-tuning foot_4 is done with 95% of the data, with the remainder used for validation. by observing the evolution of the word error rate (wer) 7 and validation loss through training, we find that the model begins to plateau half an epoch in and converges in approximately two training epochs, beyond which the model begins to overfit the dataset. training for 3,072 steps (batch size 16 and evaluation every 256 steps), we achieve a minimum wer of 1.0 at step 2,048, where validation loss is also minimised foot_6 . this checkpoint defines the fine-tuned model uesd in this study. ## dealing with hallucinations off-the-shelf, whisper requires an audio file uploaded manually in a controlled process. adapting whisper to a real-time pipeline presented an unexpected challenge: hallucinations in asr are defined as 'recognised' text that arises completely independently from what is spoken. while not limited to real-time asr @xcite , the phenomenon becomes more apparent in this adaption. hallucinations are not simply mis-recognised words or phrases, but recognition in the absence of speech. these need to be prevented as hallucinated text, while often common words/phrases, e.g. "thank you", "yes", can be unexpected or even harmful @xcite . without automatic mitigation, hallucinations may cause confusion in care records and require additional work to fix, resulting in the opposite of what we aim to achieve with our smart speaker. we find two causes of hallucination in our setting, as detailed below. averages are weighted as proportions of each accent in the test data differ, shown by the number of test samples. true wer scores (maximum precision) were included in calculation of averages, that are then rounded at the end. class imbalance is due to random sampling and reflects the number of volunteers for each accent during creation of the original dataset. ## real-time recognition results we present the results of real-time recognition with our fine-tuned model and hallucination-prevention methods in the model's full potential is demonstrated with both hallucination methods reaching a minimum wer across all tests of 6.6. the improvement over the original whisper model is substantial (down from @xmath0.4), although limited compared to the reduction achieved with fine-tuning ( some of the errors we find in the transcripts are not necessarily inaccuracies, but rather mismatches with the original script. for example, sometimes the model will recognise "uk" as "united kingdom", and "100,000s" as "hundreds of thousands". while we consider the original script as the 'ground truth' for these tests, considering these differences as correct, wer reduces to as low as 5.1 with our fine-tuned model. results for these cases are shown in without the snr checks in place, there is less reduction in performance (@xmath1.7), however, the crucial importance of including a snr threshold is demonstrated when 3-second pauses are taken after every sentence. recognition from silence/background noise results in multiple hallucinations throughout. the generated text during these quiet moments is often gibberish, repeated out multiple times and with no relation to the context of the previous speech, increasing wer to as high as 185.5. in comparison, the same test with speech pauses using both hallucination methods, achieves similar results to the first test: @xmath2.0. finally, we test our fine-tuned model in real time without checking the audio stream for repeated recognitions of overlapping data. the wer is again high at 57.8 and results in several instances where a sentence is hallucinated or repeated multiple times in the transcript. these results highlight that our fine-tuned model is more than twice as effective as whisper's original model and that hallucination prevention is essential to achieve the lowest wers possible. ## conclusion we demonstrated that an off-the-shelf whisper is not well-adapted to a wide range of spoken british accents and that wers can be reduced substantially through fine-tuning to the set of target varieties. adapting whisper as a real-time asr results in the unexpected side effect of speech hallucinations. this is addressed by enforcing a snr criterion in each audio clip and tracking audio data passed to the asr to ensure that recognised text consists of complete and accurate phrases. future work will include greater accent scope, integration into health and care plan systems, sophisticated care data querying and monitoring methods, and trigger/alert systems to improve administration efficiency and help identify errors. upon successful deployment of these features, we will trial our smart speaker in a real care-home environment to gain a better understanding of technological capabilities, user requirements and to maximise the social impact of our specialised speaker system.