A Comparative Approach for Auditing Multilingual Phonetic Transcript Archives (2025)

Abstract: Curating datasets that span multiple languages is challenging. To make the collection more scalable, researchers often incorporate one or more imperfect classifiers in the process, like language identification models. These models, however, are prone to failure, resulting in some language partitions being unreliable for downstream tasks. We introduce a statistical test, the Preference Proportion Test, for identifying such unreliable partitions. By annotating only 20 samples for a language partition, we are able to identify systematic transcription errors for 10 language partitions in a recent large multilingual transcribed audio archive, X-IPAPack (Zhu et al., 2024). We find that filtering these low-quality partitions out when training models for the downstream task of phonetic transcription brings substantial benefits, most notably a 25.7% relative improvement on transcribing recordings in out-of-distribution languages. Our work contributes an effective method for auditing multilingual audio archives.1

## introduction the development of nlp systems has been largely uneven, mostly dominated by a handful of western european languages and some non-indo-european ones @xcite . in some regards, this is improving-only in the last few years, researchers have open-sourced a number of notable multilingual archives, foot_1 such as madlad-400 @xcite , voxpopuli @xcite , and owsm @xcite . the acquisition of these large multilingual archives is complex, however. the data collec-tion pipeline needs to scale to a large number of languages and a large volume of data for each language. enabling this scaling is the use of predictive models: from language-identification models to sentence-embedding based bitext mining methods @xcite to speaker diarization models @xcite . these models, however, are prone to failure: major data collection errors in well-known archives have already been reported. in a comprehensive study over 5 major archives and over 70 languages, @xcite report systematic failures in language identification and bitext mining, with greater error rates for lower-resourced languages. further, recent studies in multilingual and long-form speech recognition found that prominent speech archives contain substantial chunks of untranscribed content, leading to a high rate of deletion errors in speech recognition models trained on this data @xcite . these studies demonstrate the complexity of acquiring high-quality multilingual data. in this light, the data collection pipeline itself can be considered an imperfect approximation of the data distribution we wish to sample from. as with any approximation, common wisdom suggests that we should aim to evaluate the quality of the approximation. unlike the wealth of empirically and theoretically established metrics and hypothesis tests for comparing two models @xcite , there is a remarkable dearth of methods for evaluating the reliability of a semi-automatically scraped dataset that may serve as ''gold-standard'' for future downstream applications. in this work, we ask: how can we efficiently identify languages or dialects where the datacollection pipeline may have failed systematically? at a high level, we want to analyze a small subset of the partition for a language from a multilingual archive, and determine whether its samples are high quality. this analysis has two important components. first, how do we determine if a sample is high quality? second, how large should the subset be to make our determination? to answer these questions, we draw on theory from statistical power analysis @xcite . specifically, we are interested in accurately estimating preference (bradley and @xcite : how much better does the partition capture the relationship of interest than an existing baseline model? in particular, we elicit preferences from human annotators to ground the partition's quality with judgments from domain experts. we ground our method in the task of phonetic transcription, where the input is recorded speech from any language, while the output is a transcription into the international phonetic alphabet (ipa). phone recognition models have important applications in language documentation, especially for oral languages @xcite . the training datasets for this task are often semi-automatically generated, so that many languages can be represented (section 2). consider the partition for one language in such a semi-automatically populated multilingual archive. if a knowledgeable user consistently prefers the output of an existing (imperfect) baseline model over the ''ground-truth'' transcripts in the partition under audit (see figure 1 ), then this indicates that this language partition is unreliable. we refer to this statistical test as the preference proportion test, or the ppt, which we introduce in section 3.2. critically, we assert that only a small fraction of examples needs to be annotated to attain a high-powered test of whether a language partition should be flagged as unreliable. to illustrate the effectiveness of the ppt, we perform a case study on a recent large multilingual phonetic transcript archive-the x-ipapack @xcite , comprising transcribed audio for 78 languages. applying the ppt, we efficiently identify 10 language partitions in the archive that have unreliable transcripts. we find that a model finetuned on the filtered version of the archive-without the unreliable 10 language subsets-generalizes better to a test set (comprising 5 held out languages) than a model trained on the complete archive. to contain a high degree of quality-control issues, especially for lower-resourced languages. we audit the quality of each language partition in a multilingual archive by annotating a small sample from it. specifically, expert annotators select between the gold-standard transcript and one generated by a baseline model prediction . when the baseline model predictions are consistently preferred over the goldstandard, we conclude that the language partition should be flagged for re-labeling. we find two ways in which low-quality data can be especially pernicious. first, we find the largest improvement on the punjabi partition of our held-out evaluation dataset (an error reduction of 20.3%) possibly due to omitting the unreliable transcripts from the sindhi partition, suggesting that the effects of poor data quality can tamper with performance in related language varieties. second, we also find a 25.4% improvement on out-of-distribution languages after training on the ppt-filtered archive, suggesting that low-quality data has a considerable impact on lower-resourced languages. our empirical results add nuance to the purported benefits of data-scaling @xcite . finally, we emphasize that filtering out lowquality data, while highly effective, is not a panacea for building robust multilingual models. some works have suggested that high-quality data on a small number of language varieties is sufficient to obtain a ''universal'' or ''languageagnostic'' model @xcite . our empirical results in phonetic transcription, however, do not support this position. leveraging a phone segment-level error metric, we find that existing universal phonetic transcription models are instead highly attuned to sounds that are more common in their respective training datasets, while making more errors on unfamiliar sounds. overall, this suggests that more diverse and highquality data collection is required for equitable performance across languages and their varieties. @xcite analyze large-scale nlp datasets through the lens of traditional archival studies. they argue that nlp datasets can be interpreted as power-laden (informal) archives. these archives not only afford the capabilities of future generative models, but-in the case of multilingual archives-also mediate our understanding of different cultures. thus, appraising the quality of different language partitions of these multilingual archives is highly important. ## case study: auditing x-ipapack in this section, we introduce the preference proportion test for efficiently auditing the quality of a multilingual archive. we ground our explanation in a case study of the x-ipapack archive. we first describe the x-ipapack archive and the preprocessing of the text in the phonetic transcripts (section 3.1). then, we introduce the test (section 3.2) and apply it to the x-ipapack archive (section 3.3). @xcite . ## data quality effect on downstream performance we now assess the downstream effect of removing the @xmath0, . . . , d l from the x-ipapack archive d. we do so by training two phone recognition models, one on d and one on d -(d 1 ∪ • • • ∪ d l ). more specifically, we finetune two whisper (small; @xcite models on these datasets. we refer to the former as whisper and the latter as whisper-ppt. further, we train an additional model, also with l partitions removed, but on partitions that were not flagged by the ppt. we randomly selected and removed tamil, xhosa, cantonese, hindi, mandarin, bengali, finnish, hungarian, italian, kyrgyz, and maori; we refer to this model whisper-anti-ppt. we provide hyperparameter details in appendix a. to contextualize our results, we also compute performance for two other models that demonstrated strong performance on multilingual phone recognition: @xcite . we use the small variant of the whisper model. the macro-average is computed by averaging the error rate across all 5 languages, with each language given equal weight. for our evaluation dataset, we select 5 language partitions from section 3.3 that passed the ppt, indicating they are trustworthy for evaluation. specifically, we evaluate on a test set d test comprising the x-ipapack evaluation partitions for thai, english, japanese, punjabi, and amharic. we hold these languages out from training of the two whisper-based phone recognition models. to evaluate our models, we again compute the pfer using phonetic feature vectors from panphon @xcite . ## dataset coverage limits downstream performance our analyses in training phone recognizers in section 3 demonstrated that poor-quality data impairs generalization. however, filtering out lowquality data from multilingual archives does not necessarily guarantee multilingual generalization. prior work had claimed universal phone recognition capacity from training on a small number of languages on high-quality data @xcite , but we demonstrate that multilingual generalization remains limited by the training data composition. the number of attested sounds across the world's languages is large @xcite , and their frequencies have a zipfian distribution (macklin-cordes and round, 2020), making them challenging to learn from limited data. we demonstrate these challenges through two phone segment-level error analyses. purported universal phone recognizers cannot transcribe clicks. since xls-r fair and xls-r nd were not trained on any bantu languages or any other language or dialect containing click consonants, we find they are incapable of predicting clicks in ( to provide finer-grained measurements of efficacy at recognizing a certain sound, we use the expected phone error (epr) metric, defined as follows. given a phone q, we identify all its occurrences in the ground-truth transcripts g(t). assuming that q appears at position i in transcript g(t), we then compute the error as the phonetic feature distance (using panphon; mortensen et al., 2016) between g(t) i and the phone in m(x) that is aligned to g(t) i . we then average the error from each occurrence of q in the dataset. we apply the needleman-wunsch algorithm for the alignment. models vary widely in the phonetic details they can capture. in ## conclusion we present the preference proportion test (ppt; section 3.2) for efficiently and systematically auditing the quality of data from specific languages in a large multilingual archive. we apply the ppt for efficiently identifying low-quality language partitions in the recent x-ipapack archive @xcite . our audit is effective, efficiently identifying language partitions whose complete removal brings substantial improvements in the downstream task of automatic phone recognition. appraising the quality of multilingual archives is critical stewardship, ensuring that they are reliable, trustworthy, and representative @xcite . overall, our method contributes an important procedure for statistically-principled multilingual archive auditing.