Hypernetworks for Personalizing ASR to Atypical Speech (2024)

Abstract: Parameter-efficient fine-tuning (PEFT) for personalizing automatic speech recognition (ASR) has recently shown promise for adapting general population models to atypical speech. However, these approaches assume a priori knowledge of the atypical speech disorder being adapted for—the diagnosis of which requires expert knowledge that is not always available. Even given this knowledge, data scarcity and high inter-/intra-speaker variability further limit the effectiveness of traditional fine-tuning. To circumvent these challenges, we first identify the minimal set of model parameters required for ASR adaptation. Our analysis of each individual parameter’s effect on adaptation performance allows us to reduce Word Error Rate (WER) by half while adapting 0.03% of all weights. Alleviating the need for cohort-specific models, we next propose the novel use of a meta-learned hypernetwork to generate highly individualized, utterance-level adaptations on-the-fly for a diverse set of atypical speech characteristics. Evaluating adaptation at the global, cohort, and individual-level, we show that hypernetworks generalize better to out-of-distribution speakers, while maintaining an overall relative WER reduction of 75.2% using 0.1% of the full parameter budget.

## introduction large-scale automatic speech recognition (asr) models are trained predominately on speech collected from the general population and historically have not been able to fully support speakers with atypical speech. recent work has proposed parameter-efficient fine-tuning (peft) of large asr models for adapting such general popula-tion models to work better for people with speech differences @xcite . such adaptations have focused either on fine-tuning using data from a group of individuals with common speech differences-referred to as cohort-level fine-tuning-or on fine-tuning on speech data at the level of an individual. individually personalized asr models yield state-of-the-art transcription performance, however they require laborious data collection, which could be especially strenuous for people with severe speech disorders. additionally, characteristics vary greatly, even for the same speaker over time, potentially leading to data drift and eventual performance degradation @xcite . at the cohort level, peft of general population models has been shown to improve transcription performance for dysarthria @xcite . while this approach reduces training data and compute requirements, it requires a priori knowledge of an individual's atypical speech category, which is not always available. as we later demonstrate, a precise diagnosis is crucial, as fine-tuning on a specific cohort does not transfer well to other types of atypical speech (section 4.5). furthermore, such solutions require discrete categorizations of individuals and do not share knowledge across cohorts, although sharing may be beneficial for individuals who express mixtures of speech differences, or between individuals with different severities of the same speech disorder. in this work, we consolidate individual-level personalization with knowledge-sharing across cohorts by proposing the use of hypernetworks to generate adaptation parameters dynamically during inference, for individualized personalization amongst a heterogeneous cohort of speech disorders. as opposed to cohort and individual-level fine-tuning, which learn fixed adaptations that are difficult to transfer across etiologies, hypernetworks leverage a meta-learning procedure that instead learns to generate adaptation parameters, conditioned on the target speaker's speech characteristics. this approach enables flexibility with respect to the adaptations applied to the asr model, as they can change depending on the individual utterance being adapted for. simultaneously, the use of a single hypernetwork instead of multiple pre-trained adaptations for each cohort or individual reduces complexity and actively promotes sharing information that is useful across different types of atypical speech. in our study, we include both phonological and fluency-related speech disorders, using speech from people with stuttering, dysarthria consistent with cerebral palsy, and parkinson disease, for which a myriad of speech differences including dysarthria and stuttering may be exhibited. stuttering includes dysfluencies, such as sound, word or phrase repetitions (''m-m-mall'', ''go go go''), prolongations (''baaall''), and audible pauses or blocks @xcite . dysarthric speech may contain differences in pronunciation, pitch, intelligibility, strain, speaking rate and volume. it is particularly challenging as the expressed characteristics depend on the etiology and individual, varying even for one speaker @xcite . for example, spastic dysarthria, commonly associated with cerebral palsy, is characterised by slow speaking rates, strained voice, and pitch breaks @xcite , whereas hypokinetic dysarthria, commonly associated with parkinson disease, is characterized by monotonous speech, varying in volume, breathiness, hoarseness, rapid repetition of phones and imprecise consonant production @xcite . towards improving asr for these speech communities, we concretely contribute: ## related work personalized asr adaptation for atypical speech is a broad yet under-explored topic. prior work mainly focuses on large-scale asr models trained on general population speech, which are subsequently fine-tuned on small datasets of atypical speech @xcite . such datasets are scarce, and even more so at the level of individual speakers, leading to overfitting and poor generalization. to overcome these challenges, past work has explored peft methods such as individually re-weighting transcription output probabilities (morales and cox, 2007), or using residual adapters @xcite to individually personalize asr models @xcite , while retaining the original model weights and only training the light-weight adapter modules. another approach leverages cohort-level transfer learning: @xcite use a two stage fine-tuning process, where the asr model is first fine-tuned on data from a cohort sharing atypical speech characteristics, before being further fine-tuned on data from an individual in the same cohort. @xcite follow a similar approach, but make use of less resourceintensive adapter fusion @xcite , where a cohort-level adapted model is fused with multiple individual-level adapters to train personalized models for new target speakers. these aforementioned studies either require sufficient data from the target speaker in order to train a personalized model, and/or knowledge of the cohort an individual belongs to-both of which may not be readily available. as we demonstrate in section 4.5, the process of maintaining and selecting the correct cohort model is critical, however defining the cohort is nontrivial as assigning membership may not be limited to etiology but also severity thereof. furthermore, prior approaches consider cohorts independently of each other and are thus unable to share knowledge that may be beneficial for better generalization performance across individuals. in order to generate individualized adaptations while learning globally shared representations across cohorts, we reformulate asr adaptation as a meta-learning problem. we propose to model this inductive bias via a light-weight hypernetwork meta-learner @xcite , which is tasked to generate the most effective adaptation weights for an individual based on their speaker characteristics as represented by a shared encoder. while this work, to the best of our knowledge, is the first to apply hypernetworks to asr, recent studies have applied them to predict the task-specific adaptations of a text-based, pre-trained, large-scale transformer architecture @xcite . additionally, language model adaptations generated by hypernetworks have also been shown to generalize to unseen task and language combinations @xcite . based on these results as well as recent successes in adapter fusion @xcite , we hypothesize that zeroshot personalization is possible by having the hypernetwork learn a mapping between speaker characteristics and asr adaptation weights, effectively learning a manifold of personalized models. this procedure would require neither labelled audio data nor fine-tuning on the individual-level, leading to increased parameter and data efficiency. ## cohort-level personalization generating an entire personalized model would be prohibitively expensive. as such we first follow the cohort-level personalization paradigm to identify which pre-trained models provide good initializations for adaptation (section 4.1), and which components (section 4.2) and individual parameters (section 4.3) are crucial to fine-tune and adapt. this allows us to create a lighter-weight adaptation framework to which we later apply dynamic personalization (section 5). ## dynamic personalization our previous findings generalize across speech disorders to support higher parameter efficiency than prior work. however, in practice, cohort-level personalization still requires knowledge of the target etiology, as a model trained on data from one cohort does not transfer well to another. additionally, severe cases of atypical speech, being rarer within the cohort, see less improvement than mild and moderate cases. therefore, a datacentric design that is more cognizant of both speech disorder type and severity is necessary for successful personalization. towards this goal, our second contribution is the design and use of light-weight hypernetworks @xcite to dynamically generate personalized adaptation weights-essentially generating a new adapted model for each utterance at inference time. ## results we next compare our proposed approach to the full, partial, and lora baselines outlined in section 3, and report the findings in tables 2 and 3 in decreasing order of heterogeneity of the training data, namely at the global, cohort, and individual level. results are reported for whisper large-v2, which represents the upper bound in terms of performance. ## conclusion due to data scarcity and the highly variable nature of atypical speech, prior work relied on cohort-level fine-tuning and peft, followed by individualized adaptation. while this approach substantially reduces wer, it necessitates training as many models as there are speakers, and requires expert knowledge of the cohort a speaker belongs to. in section 4.5, we demonstrate that this knowledge is critical, as a model trained on one cohort does not transfer to others. in addition, higher severity speakers and those with multiple pathologies still do not benefit from the improvements for mild and moderate cases, as they are least represented in the data. combining meta-level knowledge sharing and highly individualized personalization, we therefore proposed using hypernetworks to dynamically generate adaptations during inference (section 5). as generating an entire model is computationally infeasible, we first conducted a study regarding which individual model parameters have the highest influence on adaptation performance (section 4). analyzing model components at increasing levels of detail using a novel combination of lora and ssas, we identified a single parameter type w 1 which contributes most to adaptation performance. as this effect was consistent across different model sizes, pre-training strategies and lora ranks, we were able to halve wer while using 0.03% of the parameter budget required for full fine-tuning. based on these findings, we were able to scale our hypernetwork-based asr adaptation approach to an even smaller parameter budget of 0.01%. despite sharing a single model across cohorts and having access to neither cohort or individual speaker information, hypernetworks reach a wer of 4.0, consistently outperforming lora, and performing competitively to full fine-tuning (section 6.1). these results hold, even when the latter two approaches are specifically fine-tuned using in-cohort data (section 6.2) and/or training data from a target individual (section 6.3). further ablating the hypernetworks' parameter generators (section 6.4) and host model architectures (section 6.5), we find our general meta-learning approach to generalize well across both datasets and models. our analyses in sections 5.2, 5.3, and 6.4 further surface factors critical to hypernetwork performance: firstly, improving upon prior approaches, we find that nulling out initial adaptations is crucial for not deteriorating existing model performance. second, for speaker characterization, continuous coverage across acoustic features appears more important than discrete speaker featurizations. third, the hypernetwork meta-learns adaptation weights which exhibit overlaps, matching etiological intuitions beyond the information present in the input embeddings. improving upon prior work, we demonstrate that this general approach holds across multiple types of atypical speech, enabling zero-shot dynamic personalization without explicit knowledge of cohort membership, nor training data from the target speaker. in this work, we target a diverse set of dysfluency and phonology-related speech disorders. however, future work may investigate other categorizations of atypical speech that are not included in our datasets. additionally, we believe exploring a broader range of asr scenarios could be fruitful, especially since standard adapters have been shown to work well for heavy-accented speech, and multi-lingual settings @xcite . applying hypernetworks to these, as well as a myriad of other downstream scenarios, could therefore form an interesting extension to this work. lastly, our experiments centered around multiple variants of whisper, confirming our findings across different model sizes and pre-training data regimens. we believe future work could follow our general approach for identifying relevant adaptation parameters, and subsequently generate them using meta-learned hypernetworks, in order to better optimize parameter efficiency when adapting alternative backbone architectures in low-resource scenarios.