Adapting Whisper for Regional Dialects: Enhancing Public Services for Vulnerable Populations in the U nited K ingdom (2025)

Abstract: We collect novel data in the public service domain to evaluate the capability of the state-of-the-art automatic speech recognition (ASR) models in capturing regional differences in accents in the United Kingdom (UK), specifically focusing on two accents from Scotland with distinct dialects. This study addresses real-world problems where biased ASR models can lead to miscommunication in public services, disadvantaging individuals with regional accents particularly those in vulnerable populations. We first examine the out-of-the-box performance of the Whisper large-v3 model on a baseline dataset and our data. We then explore the impact of fine-tuning Whisper on the performance in the two UK regions and investigate the effectiveness of existing model evaluation techniques for our real-world application through manual inspection of model errors. We observe that the Whisper model has a higher word error rate (WER) on our test datasets compared to the baseline data and fine-tuning on a given data improves performance on the test dataset with the same domain and accent. The fine-tuned models also appear to show improved performance when applied to the test data outside of the region it was trained on suggesting that fine-tuned models may be transferable within parts of the UK. Our manual analysis of model outputs reveals the benefits and drawbacks of using WER as an evaluation metric and fine-tuning to adapt to regional dialects.

## introduction automatic speech recognition (asr) systems are becoming increasingly embedded in our technologies and processes @xcite . the ease of use of these systems (ibrahim and varol, 2020) combined with recent advancements in performance with the use of more sophisticated models makes it particularly appealing for domains with * equal contribution. limited resources, including legal areas @xcite , healthcare @xcite and other public services. as a result, it is important to address potential problems, particularly those that amplify sociolinguistic biases. regional and social dialects resulting in speech of the same language having phonological, lexical and grammatical differences present significant challenges for asr systems @xcite . as english is a high-resource language, there are copious amounts of data available to train asr models to recognise english. despite this, many models struggle with variations and dialects of english that are underrepresented in training data @xcite . this phenomenon is observed for multiple variations of english including decreased performance for african american vernacular english @xcite , english as a second language or non-native english @xcite and variations of english within regions including the uk @xcite . the lack of inclusivity in asr often leads to disparities between users of these systems @xcite . as a result, in this work, we investigate the performance of asr systems on regions in the united kingdom (uk), specifically areas where accents are less commonly represented in speech datasets. the uk also has socioeconomic links to accent @xcite @xcite . this is something that may be observed in other countries across languages and so we hope this work will be transferable beyond just english in the uk @xcite . this research focuses on the state-of-the-art model whisper @xcite , a multilingual asr system that is increasingly used in industry settings. whisper is trained on a diverse set of 680,000 hours of multilingual data, making it particularly robust for recognising speech across languages, including those with less train-ing data. whisper is designed to handle real-world audio with noise and challenging conditions better than many existing asr models. the model demonstrates lower word error rates (wer) compared to earlier models across a variety of benchmarks, including librispeech, common voice, and other multilingual datasets @xcite ). whisper's performance gains have been validated through community usage and industry adoption in particular has motivated the choice to investigate and assess whisper's capabilities. in this work, we explore whisper's capabilities to recognise accented speech in public service settings in two areas of the uk, south east scotland and north east scotland. ## data we collect new data to assess whisper's performance on a real-world use case of call transcription. the collected data represents two groups of accents from the uk and consists of calls from two public service scenarios. the real names of these organisations we collect data from have been omitted throughout the paper and replaced with the following representative terminology: north east scotland advice charity (nesac) and south east scotland housing association (sesha). these charities provide critical services to the community particularly in vulnerable populations, with a large proportion of callers likely coming from low socio-economic backgrounds vitally in need of these services. nesac and sesha offer free legal advice and housing support, respectively, making accurate transcription essential for effective communication and service delivery. both charities are located in areas with different dialects situated in scotland. the datasets have been manually annotated with accent labels and manually transcribed for training and comparison with the machine generated transcriptions, we refer to this as the "human transcript". we use a subset of our collected data for fine-tuning and the remaining data is reserved for testing. additionally, we use the open-source multi-speaker corpora of the english accents in the british isles dataset @xcite as a baseline dataset for all models. ## experimental setup to address the research questions outlined in section 1.1. we run two experiments and a manual analysis. the first experiment looks at the effectiveness of whisper in capturing variations in dialect in the uk and the second explores fine-tuning as a mechanism to adapt the whisper model to accents. finally, we conduct a manual analysis of model errors to better understand the effectiveness of our chosen evaluation metric wer. this section describes the experimental setup for these experiments. we test the whisper large-v3 model on a subset of our nesac and sesha datasets where each test set has approximately 5 hours of data. the large-v3 model for whisper was selected over the other sizes available as it gave the best performance in our initial experiments. whisper large-v3 is also used as a base model in our fine-tuning experiment. we fine-tune two models, one using nesac and the other using sesha. the same two test sets from the first experiment are used to evaluate the performance of the finetuned models as the training and test data were separated before fine-tuning. for the training of the fine-tuned models a learning rate of 5x10 -6 and a batchsize of 64 were used with 47 hours of the nesac data used to train the nesac fine-tuned model and 46 hours of the sesha data to train the sesha fine-tuned model. ## experiment 1: whisper to answer research question 1 outlined in section 1.1, this experiment focuses on the out-of-the-box performance of the whisper large-v3 model on our collected data representing accents from north east scotland captured in nesac and south east scotland captured in sesha. the results of this experiment are shown in figure 1 and the first row of ## experiment 2: fine-tuned models ## manual analysis to address research question 3 outlined in section 1.1 and better understand the effectiveness of wer as an evaluation metric for our models, we manually inspect a portion of the errors from each model on the baseline data as well as the nesac and sesha test data. since the nesac and sesha datasets contain sensitive information, we mostly present our findings with examples from the baseline dataset. although the fine-tuned models exhibited higher wer on the baseline data compared to the whisper large model, our manual analysis suggests that this does not necessarily indicate a worse performance. ## conclusion and future work this work uses a novel dataset to assess whisper's ability to recognise speech from two dialects in the uk. we evaluate whisper large and fine-tuned versions of the model on a baseline dataset and our two test datasets. we find that all of the models have worse performance on our north east scottish and south east scottish test data compared to the baseline data, the whisper model performs better when it is fine-tuned and tested on data from the same distribution and there may be evidence of dialect transferability for our fine-tuned models. we conducted a manual analysis of the errors from each model and found that differences in transcription style appear to negatively impact the observed wer. the manual analysis also demonstrated evidence of the fine-tuned models successfully adapting to the target dialect as well as cases where the fine-tuning approach negatively impacted the models' contextual understanding. this indicates the need for a careful balance during the fine-tuning process and highlights both the potential and the drawback of using fine-tuning for variations in english in public services for vulnerable populations. we hope to investigate the transferability of finetuned whisper models further in future work by collecting more data that represents a wider range of accents from within the uk and evaluate the transferability of fine-tuned models on accents from these other regions. furthermore, we aim to incorporate approaches that avoid the use of confidential and sensitive data, which nesac and sesha are in this case.