Enenlhet as a case-study to investigate ASR model generalizability for language documentation (2024)

Abstract: Although both linguists and language community members recognize the potential utility of automatic speech recognition (ASR) for documentation, one of the obstacles to using these technologies is the scarcity of data necessary to train effective systems. Recent advances in ASR, particularly the ability to fine-tune large multilingual acoustic models to small amounts of data from a new language, have demonstrated the potential of ASR for transcription. However, many proof-of-concept demonstrations of ASR in low-resource settings rely on a single data collection project, which may yield models that are biased toward that particular data scenario, whether in content, recording quality, transcription conventions, or speaker population. In this paper, we investigate the performance of two state-of-the art ASR architectures for fine-tuning acoustic models to small speech datasets with the goal of transcribing recordings of Enenlhet, an endangered Indigenous language spoken in South America. Our results suggest that while ASR offers utility for generating first-pass transcriptions of speech collected in the course of linguistic fieldwork, individual vocabulary diversity and data quality have an outsized impact on ASR accuracy.

## introduction the fields of descriptive and documentary linguistics concentrate on collecting and analyzing language samples, particularly from understudied, indigenous, and endangered languages. typically, documentary linguists -who can be researchers or language community members -make audio recording of unscripted or prompted speech, followed by transcription, glossing, translation, and analysis. transcription, however, often becomes bottleneck when dealing with large speech corpora, rendering only a fraction of the available speech data available for analysis or for language instruction @xcite . automatic speech recognition (asr) has emerged as a potential solution by providing firstpass transcripts that can be manually corrected @xcite @xcite . the preferred approach for building an asr system with scarce resources is to fine-tune a large multilingual model to whatever small amount of transcribed audio data is available for the target language. many demonstrations of the efficacy of this approach, however, rely on corpora with relatively few speakers or with recordings made under the same condition (e.g., all read speech or broadcast news) @xcite . one problem is that models trained on a single uniform speech corpus may overfit to that corpus acoustically or lexically. it is not clear how such models will generalize to new data -whether that data is archival recordings or recent data from a different speaker population or data collected with different prompts. in this paper, we address this question using a corpus of speech recordings for the language enenlhet (iso 639-3 code tmf; not to be confused with the related language enlhet, iso 639-3 code enl). enenlhet is spoken by fewer than 2,000 people living in what is now paraguay. while thousands of the world's languages, like enenlhet, are endangered and have minimal written documentation, many of these languages lack three important characteristics that make enenlhet an ideal language for exploring the utility of asr for documentation of diverse speech data. first, the amount of available transcribed speech data -5 hours -is relatively substantial for an endangered and primarily oral language. second, the large quantity of untranscribed audio -over 100 hours -is highly unusual for any endangered language, offering the potential for unsupervised training and for experimentation with integrating asr into the documentation pipeline. third, the enenlhet speakers who provided their voices have stated enthusiasm for generating new 1 132 documentation for their language in collaboration with outsiders. using two asr architectures that support finetuning acoustic models to the task of asr for small speech datasets, we explore to what extent an existing corpus can be used to train models that generalize well to new data. the dataset we use, while part of a single data collection effort, was collected over multiple years under varying conditions from a large number of speakers. we simulate introducing a new recording by holding out each speaker in turn, training on the remaining speakers, and testing on the held-out speaker. we find that the high degree of lexical diversity across speakers, as well as differences in audio and transcription quality, contribute to variability in word error rate (wer), findings we quantify with a regression analysis. ## data the language of the corpus used in this study is known by a number of names (cabanatit, enenxet, toba-enenlhet), but following the preferences of the community, we will refer to it here as enenlhet (iso 639-3 code tmf). enenlhet is a polysynthetic language spoken by fewer than 2,000 people living in the paraguayan chaco region. migration and displacement have led to dramatic language loss; the current ethnologue status of enenlhet is 6b (threatened). enenlhet has remarkably little available documentation. aside from a few short word lists compiled in the 1920s and 1960, there are no dictionaries, and there is only one available grammar @xcite . a phone set of approximately 4 vowels and 15 consonants can be inferred from the dataset described below. the data used here is part of a recent multi-year data collection effort, which has so far yielded over 120 hours of recordings with more than 40 individuals. the data was collected with university irb approval and is archived with the archive of indigenous languages of latin america at the university of texas. ethical practices require a consultation with the language community before using language material for research purposes @xcite . thus, while the data we use is publicly available 1 , the co-author who collected the data gained express permission from the community for research purposes. she notes that the enenlhet speakers who participated in the data collection were eager for their speech recordings to be used to support documentation and revitalization efforts. approximately 10 hours of the recordings from 16 speakers have been transcribed with utterancelevel timestamps. the total quantity of speech data available in these recordings -after stripping out silence, segments in another language, and speech produced by the interviewer -is approximately 5 hours. ## results results are presented in figure 2 . each bar corresponds to a test conducted on a different speaker's data. the baselines are indicated by the dotted lines. first, we see across all scenarios, wav2vec performs systematically worse than whisper. second, we observe in all experiments, the baseline does not exhibit consistent inferior or superior performance in either architecture. we note that potential biases during experiments conducted on random splits do not significantly impact overall performance in one the results of the regression analysis can be found in two metrics were used evaluate the quality of the audio in the test sets. loudness was found to have minimal impact, but sharpness appears to negatively impact wer. additional experiments on larger datasets are necessary to validate the efficacy of this measure in assessing audio quality. the posterior probabilities obtained from the ctc alignment exhibited a strong negative impact on wer, suggesting that a higher confidence score in the alignment corresponds to a lower wer. however, the confidence interval is relatively high, raising doubts about the reliability of this measure to evaluate the quality of the transcription. examining specific examples, we verified the data quality for if, where a very high wer was observed. it was discovered that the transcription for this speaker did not align with the audio; instead, it appeared to be a translation of the audio into a related language or dialect. however, the ctc alignment did not substantially differ from other speakers where the transcription matched well and the wer is much lower. this measure appears instead to be relevant for evaluating audio quality when there is not a significant mismatch with the transcription. ## conclusions this paper explores how contemporary speech recognition architectures perform in a language documentation setting, focusing on the enenlhet language as a case study. in order to simulate testing of new data using a model trained on a previous data collection corpus, we conducted training and testing of asr models using a leave-one-out evaluation approach, where the models were trained on all enenlhet speakers except one and tested on the one left out. additionally, we performed a re-4 135 gression analysis to determine the factors that may influence wer. the experimental results initially revealed that the leave-one-out evaluation approach neither outperforms or underperforms a random split approach for our specific case. subsequently, we found that out-of-vocabulary (oov) rates are the most significant factor in explaining the wer for a given test set. lastly, both the sharpness measure and the ctc posterior probabilities show promise in assessing the quality of the speech signal, which could potentially correlate with the word error rates. further analysis is necessary to confirm this correlation. these results suggest that in low-resource settings, asr models may not always generalize well to new data, which could hamper the utility of asr for language documentation.