Investigating data partitioning strategies for crosslinguistic low-resource ASR evaluation (2023)

Abstract: Many automatic speech recognition (ASR) data sets include a single pre-defined test set consisting of one or more speakers whose speech never appears in the training set. This “hold-speaker(s)-out” data partitioning strategy, however, may not be ideal for data sets in which the number of speakers is very small. This study investigates ten different data split methods for five languages with minimal ASR training resources. We find that (1) model performance varies greatly depending on which speaker is selected for testing; (2) the average word error rate (WER) across all held-out speakers is comparable not only to the average WER over multiple random splits but also to any given individual random split; (3) WER is also generally comparable when the data is split heuristically or adversarially; (4) utterance duration and intensity are comparatively more predictive factors of variability regardless of the data split. These results suggest that the widely used hold-speakers-out approach to ASR data partitioning can yield results that do not reflect model performance on unseen data or speakers. Random splits can yield more reliable and generalizable estimates when facing data sparsity.

## introduction certain model evaluation practices are considered standard or quite common in natural language processing (nlp), such as using popular benchmarks @xcite , pre-defined data partitions @xcite , or random splits @xcite . all of these practices rely on metrics calculated over test sets as indices of model performance. it is not generally acknowledged that a particular numerical result might be meaningful only for the specific train/test split that produced that result. a single aggregated metric does not necessarily paint the full picture of a model architecture's potential @xcite . automatic speech recognition (asr) provides a case in point. given a data set produced by multiple speakers, the common data partitioning strategy is "hold speaker(s) out", namely holding out all utterances from one or more speakers @xcite as the test set, with the utterances from the remaining speakers serving as the training set. cross-validation is generally not applied; the speakers in the test set are fixed. in other words, an asr system is usually evaluated with just a single train/test split in which there is no speaker overlap between the training and test sets. this common data partitioning strategy might fare well with a large data set, with recordings of dozens or hundreds of speakers, where the quantity of data and the wide array of speakers enable the training of models that are assumed to be speaker independent. the same practice, however, is not ideal for low-resource scenarios, where the number of speakers is much smaller. with endangered languages @xcite in particular, there is much less flexibility in deciding how much data and what kind of utterances to include. thus, observed asr accuracy may depend heavily on which speakers appear in the test sets rather than being representative of the model architecture's general performance. this study investigates alternative data partitioning methods for low-resource asr. leveraging data from five typologically distinct languages, including one endangered language, we ask: (1) how dependent is asr performance on the identity of the held-out speaker? (2) can alternative data partitioning strategies yield less variable estimates of a model's generalizability? (3) what factors other than speaker identity contribute to differences in model performance? (4) how can we operationalize lessons learned to improve asr evaluation for under-resourced and endangered languages? ## data descriptions we used data sets for four widely spoken lowresource languages, fongbe @xcite , wolof @xcite , swahili @xcite , and iban (juan et al., 2014), which were previously released as asr corpora. they include segmented audio with corresponding transcripts, as well as additional written texts for training the language model (see in addition, we explored a data set of hupa, a critically endangered language indigenous to north america. the audio recordings for hupa are the product of ongoing linguistic fieldwork started in 2005. all the recordings were produced by a single female elder speaker, which is common for speech corpora for critically endangered languages, mak-ing hupa a unique test bed for our study. each transcription typically goes through several stages of correction and consultation with the elder before being considered complete; thus some transcriptions have been examined more thoroughly than others. based only on differences in transcription quality, the audio data was divided into two sets, which we will call "verified" vs. "coarse" data respectively (details are presented in appendix a.1). ## results we first consider the degree of variability in wer depending on which speaker is held out. in figure 1 , for fongbe, wolof, and iban, we see a high degree of variability. the wer range across heldout speakers spans from 12.71 for iban to 54.74 for fongbe. (see one might suspect that the observed wer variability across speakers in each data set is (only) because of the varying amount of audio available for each speaker. although there is a relationship between average wer and the total utterance duration per speaker (when looking at total utterance duration as a sole predictor in the regression) for iban (@xmath0.005), this relationship does not hold for fongbe (@xmath1.75) or wolof (@xmath2.91). while there is a positive correlation between duration and wer for the coarse data of hupa (@xmath3.05), this correlation does not exist for swahili (@xmath4.45) or for the verified data of hupa (@xmath5.99). this indicates that the total utterance duration alone is not enough to yield (high) wer variability across speakers. in contrast, results from random splits are much less variable. while not surprising, this is notable in that the average wer when holding out a speaker or session is comparable to that of random splits, or any one random split. thus a single random split can alone be enough to provide a reasonable estimate of the performance that would be derived by averaging over all random splits or over all possible held-out speakers/sessions. in contrast, the wer for a model tested on single held-out speaker/session may not be a reliable estimate of the wer of that model on any other speaker. on the other hand, splitting data heuristically and adversarially, creating test sets that consist of "more challenging" cases than the training sets, does not necessarily lead to higher wer. the results across the data split methods are mostly comparable except for when utterance duration or ## discussion and conclusion with data for four widely-spoken low-resource languages and one critically endangered language indigenous to north america, our work demonstrates that there is a real risk of grossly over-or underestimating the performance of an asr model architecture when evaluating on held-out speakers (and sessions) when only minimal resources are available. by contrast, random splits provide a more accurate and less variable estimate of the overall performance. moreover, while cross-validation is advisable when partitioning by speaker, a single random split appears to provide an adequate estimate for expected wer on unseen data. we note that these findings also hold for data sets partitioned according to recording session rather than speaker, suggesting that this phenomenon is not limited to diversity in speaker characteristics. this has implications particularly for asr in support of endangered language documentation, in which the number of speakers is few but the recording conditions are highly variable. we propose that future work on small asr corpora for underresourced languages carry out multiple evaluations on various data partitioning strategies in order to present a more complete picture of asr model architecture performance.