A ccent F old: A Journey through A frican Accents for Zero-Shot ASR Adaptation to Target Accents (2024)

Abstract: Despite advancements in speech recognition, accented speech remains challenging. While previous approaches have focused on modeling techniques or creating accented speech datasets, gathering sufficient data for the multitude of accents, particularly in the African context, remains impractical due to their sheer diversity and associated budget constraints. To address these challenges, we propose AccentFold, a method that exploits spatial relationships between learned accent embeddings to improve downstream Automatic Speech Recognition (ASR). Our exploratory analysis of speech embeddings representing 100+ African accents reveals interesting spatial accent relationships highlighting geographic and genealogical similarities, capturing consistent phonological, and morphological regularities, all learned empirically from speech. Furthermore, we discover accent relationships previously uncharacterized by the Ethnologue. Through empirical evaluation, we demonstrate the effectiveness of AccentFold by showing that, for out-of-distribution (OOD) accents, sampling accent subsets for training based on AccentFold information outperforms strong baselines a relative WER improvement of 4.6%. AccentFold presents a promising approach for improving ASR performance on accented speech, particularly in the context of African accents, where data scarcity and budget constraints pose significant challenges. Our findings emphasize the potential of leveraging linguistic relationships to improve zero-shot ASR adaptation to target accents.

## introduction english language is spoken in 88 countries and territories as either an official, administrative, or cultural language, estimated at over 2 billion speakers with non-native speakers outnumbering native speakers by a ratio of 3:1. despite considerable advancements, automatic speech recognition (asr) technology still faces challenges with accented speech @xcite @xcite . speakers whose first language (l1) is not english have high word error rate for their audio samples @xcite . @xcite showed that existing asr systems struggle with speakers of african american vernacular english (aave) when compared with speech from rural white californians. the dominant methods for improving speech recognition for accented speech have conventionally involved modeling techniques and algorithmic enhancements such as multitask learning @xcite @xcite , domain adversarial training @xcite , active learning @xcite , and weak supervision @xcite . despite some progress in asr performance, performance still degrades significantly for out-of-distribution (ood) accents, making the application of these techniques in real-world scenarios challenging. to enhance generalizability, datasets that incorporate accented speech have been developed @xcite . however, given the sheer number of accents, it is currently infeasible to obtain a sufficient amount of data that comprehensively covers each distinct accent. in contrast, there has been a relatively smaller focus on exploring linguistic aspects, accent relationships, and harnessing that knowledge to enhance asr performance. previous research in language modeling @xcite , intent classification @xcite and speech recognition @xcite @xcite have demonstrated that incorpo-rating linguistic information in nlp tasks generally yields downstream improvements, especially for languages with limited resources and restricted data availability -a situation pertinent to african languages. consequently, we opine that a deeper understanding of geographical and linguistic similarities, encompassing syntactic, phonological, and morphological aspects, among different accents can potentially enhance asr for accented speech. we believe embeddings offer a principled and quantitative approach to investigate linguistic, geographic and other global connections @xcite , and form the framework of our paper. our contribution involves the development of accentfold, a network of learned accent embeddings through which we explore possible linguistic and geographic relationships among african accents. we report the insights from our linguistic analysis in section 4. by conducting empirical analysis, we demonstrate the informative nature and practical significance of the the accent folds. concretely, in section 5, we show that for a given target ood accent, fine-tuning on a dataset generated from a subset of accents obtained through accentfold leads to improved performance compared to strong baselines. ## related work using existing state-of-art pre-trained models to probe for linguistic information and using that to improve models' performance has gained interest in the community recently. prasad and jyothi (2020) use various probing techniques on the deep-speech 2 model @xcite . they find that first few layers encode most of the accent related information. @xcite quantify language variation in dutch using a combination of xls-53 @xcite embeddings and dynamic time warping @xcite . they show that this leads to a dutch dialect identification system that is better than a system dependent on the phonetic transcriptions with just six seconds of speech. thus, proving that pre-trained models such as the one proposed by @xcite indeed capture rich linguistic information in their representations. @xcite ; @xcite extract accent embeddings learnt from a separate network and input those embeddings along with other features. they show that this leads to a superior accented asr model. our work is most closely related to @xcite , where the authors explore various statistical methods such as submodular mutual information in combination with hand-crafted features to select a subset of data to improve accented asr. our work differs from previous works in two important ways (1) we take a different approach and use the extracted accent embeddings from a pre-trained model to decide what subset of data to use to build an asr that performs the best on a target accent in a cost-effective manner (2) we do this at a much larger scale of 41 african english accents. note that the previous highest was 21 english accents by @xcite . ## accentfold this section outlines the procedures involved in the development of accentfold. ## what information does accentfold capture? in this section, we delve into an exploratory analysis of the t-sne visualizations for all the accents in accentfold. our aim is to gain a deep understanding of the intricate connections and patterns that emerge among these diverse accents. the t-sne visualizations of the accent in accentfold can be found in figures 2 , 3 , 4. we also present some more figures (8, 9, 10, 11) in the appendix. language families: figure 10 presents a t-sne visualization of the learned accent embeddings, where color coding is utilized to distinguish language families, and varying levels of transparency ensure distinct colors for each accent. each point in the figure corresponds to an accent embedding obtained through accentfold, allowing us to convey two pieces of information: the distribution of accents and their respective language families. through an exploratory analysis of figure 10 , we observe that the accent embeddings tend to group together (forming what we refer to as "accent folds") based on language family similarities. language families represent the genetic connections between languages, as they consist of languages that descended from a common ancestor @xcite . these language families exhibit syntactic, phonological, and morphological relationships (de marneffe and nivre, 2019). based on these observations, we hypothesize that accent-fold captures linguistic regularities within accents. geographically consistent clusters: although the majority of the data comes from nigeria, figure 3 plots all test samples with their country labels showing spatial relationships between countries. the t-sne plots generally align with geographical disposition, accents from nigeria (orange) are closer in vector space to ghana (blue) but further from kenya, uganda, rwanda, and south africa likely reflecting the distinct languages spoken across these countries. however, where similar languages (e.g. swahili) are spoken across countries (e.g. botswana and south africa), the spatial distinction is less apparent. uganda, kenya, and tanzania cluster together while botswana and south africa cluster together and rwandan embeddings fall between both regions. this demonstrates that the learned embeddings do encode some geographical information extracted entirely from speech and accent labels. ## conclusion in conclusion, our research addresses the challenge of speech recognition for african accented speech by exploring the linguistic relationships of accent embeddings obtained through accentfold. our exploratory analysis of accentfold provides insights into the spatial relationships between accents and reveals that accent embeddings group together based on geographic and language family similarities, capturing phonological, and morphological regularities based on language families. furthermore, we reveal, in section 4.1, two interesting relationships in some african accents that have been uncharacterized by the ethnologue. our experimental setup demonstrates the practicality of accentfold as an accent subset selection method for adapting asr models to targeted accents. with a wer improvement of 3.5%, accentfold presents a promising approach for improving asr performance on accented speech, particularly in the context of african accents, where data scarcity and budget constraints pose significant challenges. our research paves the way for a deeper understanding of accent diversity and linguistic affiliations, thereby opening new avenues for leveraging linguistic knowledge in adapting asr systems to target accents.