DITTO : Data-efficient and Fair Targeted Subset Selection for ASR Accent Adaptation (2023)

Abstract: State-of-the-art Automatic Speech Recognition (ASR) systems are known to exhibit disparate performance on varying speech accents. To improve performance on a specific target accent, a commonly adopted solution is to finetune the ASR model using accent-specific labeled speech. However, acquiring large amounts of labeled speech for specific target accents is challenging. Choosing an informative subset of speech samples that are most representative of the target accents becomes important for effective ASR finetuning. To address this problem, we propose DITTO (Data-efficient and faIr Targeted subseT selectiOn that uses Submodular Mutual Information (SMI) functions as acquisition functions to find the most informative set of utterances matching a target accent within a fixed budget. An important feature of DITTO is that it supports fair targeting for multiple accents, i.e. it can automatically select representative data points from multiple accents when the ASR model needs to perform well on more than one accent. We show that compared to other speech selection methods, DITTO is 3-5 times as label-efficient for its improvements on the Indic-TTS and L2 datasets.

## introduction state-of-the-art speech recognition systems have seen tremendous progress in the last few years, with end-to-end architectures becoming a default modeling choice. while end-to-end models yield impressive word error rates (wers) and work well for certain user populations @xcite , they severely underperform when confronted with out-of-domain test utterances in target accents that are unseen or rarely seen during training @xcite . a common solution @xcite to address such mismatched settings is to adapt a well-trained, speaker-independent asr * equal contribution. model with a small amount of accent-specific target data to adapt models to the target setting. while these works propose different fine-tuning schedules that would be most beneficial given the limited amount of target data, the question of which utterances should be chosen in order to be transcribed and further used for fine-tuning has received far less attention. this is extremely important, since procuring and labeling accent-specific data is challenging and expensive. awasthi et. al. @xcite present a method to select sentences within a fixed budget that are most likely to induce asr errors to record accented audio on, resulting in higher-quality personalized asr models for target accents compared to random selection. however, they assume access to a small seed set of labeled utterances from the target speaker. we address a more realistic setting wherein we have access only to a limited number of unlabeled utterances from the target domain, and without access to accented speakers to read out the selected texts. ## related work a number of works have studied subset selection for speech recognition. @xcite use submodular function-based subset selection on generated transcripts to find a minimal set of asr training data and @xcite use an entropy measure for the same. @xcite employ a joint kullback-leibler divergencebased subset selection on out-of-domain samples for asr adaptation across acoustic characteristics such as speaker, noise and recording devices. similarly, @xcite study subset selection to obtain low-vocabulary speech corpora for asr, while @xcite use a submodular approach for data selection in machine translation. many recent papers @xcite have studied uncertainty and gradient based approaches for active learning to reduce the transcription time for asr models, while @xcite use a committee-based active learning method to select speech utterances. a number of approaches have studied adaptation to atypical speech patterns like accented and dysarthic speech, such as @xcite and @xcite which fine-tune a subset of layers using labeled data from targeted accents. @xcite employ domain adversarial training to adapt across accents. @xcite tries addressing a problem that corresponds exactly to the reverse of our setting by trying to determine the sentences a model is most error-prone to, and recording utterances for them. while this can be effective for user-driven personalization, our method is suited to settings in which we have fixed speech utterances, and the only actionable item for us is to transcribe a subset of them. all these approaches need data specifically from the target domain to be labeled toward use for training/fine-tuning. finally, a number of recent works on data selection have leveraged the submodular mutual information functions used in this work for targeted subset selection. @xcite employ the smi functions for query focused and privacy-preserving summarization, while @xcite utilize the smi functions for improving the model performance on targeted slices. recently, @xcite proposed an active learning approach using the smi functions for rare classes, redundancy, and ood data. ## submodular mutual information (smi) functions submodular functions: we let u denote the ground-set of n data points @xmath0, 2, 3, ..., n} and a set function f : 2 u -→ ℜ. the function f is submodular @xcite if it satisfies the diminishing marginal returns, namely submodularity ensures that a greedy algorithm achieves bounded approximation factor when maximized @xcite . submodular mutual information (smi): given a set of items s, t ⊆ u, the submodular mutual information (smi) @xcite is defined as i f (s; t ) = f (s) + f (t )f (s ∪ t ). intuitively, this function measures the similarity between t and s and we refer to t as the targeted set. in the setting considered in this paper, the set t (target set, also called query set) consists of a small set of unlabeled utterances from an accent, and u is a large unlabeled set of utterances from multiple accents. to find an optimal subset given a target set t , we can define g t (s) = i f (s; t ), s ⊆ u and maximize the same. using a greedy algorithm, these submodular functions can be efficiently optimized within an approximation factor (1-1/e) of the global maximum. ## ditto: our data-efficient and fair in this section, we discuss ditto our data-efficient and fair targeted subset selection method for asr accent adaptation. we show that ditto can select fair and target-relevant subsets, which is critical for fine-tuning asr models on one or more accents. the main idea of our method is to instantiate a submodular mutual information (smi) function using appropriate similarity kernels in order to jointly optimize it for targeting and fairness. we summarize our method in algorithm 1 and illustrate it in fig. 1 . concretely, we are provided a few unlabeled utterances from the accent (a target set t ) which we would like the asr model m to be adapted to. the goal is to select the most informative subset s with respect to a target t from a large corpus u of unlabeled data, called the ground set. we are given a budget constraint, which is a constraint on the total time of the selected utterances. this corresponds to the transcription budget, since the selected utterances need to be later transcribed by a human. we begin with extracting accent feature representations of the unlabeled set u and the target set t ; we subsequently discuss the feature ## experimental procedure and results we use a transcription budget of 20.5 minutes for single-accent targeting and 41 minutes when an accent pair is targeted. the average uttterance durations are 4.92s in indictts and 3.6s in l2-arctic, thus these budgets come out to 250 and 500 samples on indictts and 340 and 780 samples on l2-arctic respectively. in our proposed method, we use the approach outlined in algorithm 1, with the smi function i f set as one of the flmi or gcmi functions. we consider them since they are computationally efficient (see sec. 4), and model different characteristics in their selections. as discussed in sec. 3.1, gcmi models only query-relevance, while flmi models both query-relevance and diversity. as we shall see in sec. 6.2 and sec. 6.3, gcmi is an apt choice for targeting in some scenarios, whereas flmi outperforms all methods when fairness and diversity are to be jointly modeled with targeting. ## conclusion in this work, we propose ditto, a data efficient and fair targeted subset selection method for asr accent adaptation. ditto utilizes submodular mutual information (smi) functions to find representative speech utterances that belong to the target accent within a limit budget. we show that smi functions consistently outperform other methods for targeting. we also demonstrate that ditto is capable of targeting multiple accents fairly, which can be beneficial for deploying asr models in regions with populations that speak in more than one accent.