Comparing K aldi-Based Pipeline Elpis and Whisper for Čakavian Transcription (2024)

Abstract: Automatic speech recognition (ASR) has the potential to accelerate the documentation of endangered languages, but the dearth of resources poses a major obstacle. Čakavian, an endangered variety spoken primarily in Croatia, is a case in point, lacking transcription tools that could aid documentation efforts. We compare training a new ASR model on a limited dataset using the Kaldi-based ASR pipeline Elpis to using the same dataset to adapt the transformer-based pretrained multilingual model Whisper, to determine which is more practical in the documentation context. Results show that Whisper outperformed Elpis, achieving the lowest average Word Error Rate (WER) of 57.3% and median WER of 35.48%. While Elpis offers a less computationally expensive model and friendlier user experience, Whisper appears better at adapting to our collected Čakavian data.

## introduction the low-resource nature of language documentation challenges the capabilities of current asr tools due to a lack of pretrained language models @xcite . this challenge becomes greater when the linguistic context exhibits a high degree of variation, including code-switching. čakavian, an endangered (egids 6b) language with approximately 50,000 total speakers @xcite , represents one such situation. while traditionally considered a dialect of croatian, it differs substantially from standard croatian and colloquial štokavian varieties spoken by the majority of the croatian population. 1 in addition to differences in phonology, morphology, and syntax, the čakavian lexicon includes many borrowings from romance as well as a number of forms of * these authors contributed equally to this work and share first authorship. slavic origin that are not typical for other croatian varieties @xcite . see elpis offers a locally executable pipeline to train new asr models using kaldi @xcite . gmm-based systems like elpis are less computationally demanding and they require relatively less training data compared to neural networks. this is crucial for language documentation because pretrained tools rarely exist. given that field linguists often lack the necessary expertise and access to high-powered computational resources, complexity is an important factor. conversely, the demands of large multilingual asr models could prove justifiable if they can generalize to new contexts. @xcite . the transformer-based multilingual asr model whisper can be adapted with user data from any language. the base model includes at least 91 hours of unspecified croatian data, which almost certainly does not include čakavian speech, due to the lack of available resources. while we argue that čakavian is distinct from croa-tian 2 , the availability of a related model means linguists need not train a neural network from scratch. we utilize whisper large-v3. @xcite . our results find that the best performing fine-tuned whisper model 3 is able to outperform elpis on a sample of 3 čakavian interviews, achieving an average wer of 57.3% and median wer of 35.48%, while elpis achieved an average wer of 130.1% and median wer of 129%. ## data and methods to compare model creation using elpis and model adaptation using whisper, 5.7 hours of audio data were used for training. this included five interviews of different native speakers of čakavian and one audiobook of a čakavian translation of the little prince @xcite . ## results as shown in figure 1 , the wer distributions of elpis, each fine-tuned whisper model, and the base whisper model are shown. for all models, "whisper-ckm-3" achieved the lowest average wer of 57.3% and a median wer of 35.48% in the forced-aligned wer evaluation. the median in this context refers to the error for each 20-second transcription segment obtained in the transcription of the test interviews during evaluation. the average wer for all test interview data combined was 57.3%. ## discussion our results show that the best-performing model was obtained by adapting whisper-large-v3 using transcribed interview data. it achieved an average wer of 57.3% and a median wer of 35.48%. overall, this level of performance is still poor, but the automated transcriptions contain many segments that are largely or completely error-free. we have found in practice that some transcribers can use them successfully as guides to accelerate manual transcription. further, adapted whisper models can be shared freely online allowing other researchers to benefit from these documentation efforts. the ability to save and reuse a trained model with elpis is not transparent and represents a current drawback for the pipeline. although the wer for our čakavian asr model created with elpis was higher than previously reported wer on other languages @xcite , the system has the advantages of not requiring a pre-trained language model and the underlying technology demands fewer computational resources for im- our attempts at fine-tuning the whisper large-v3 model show the effect of several factors on model performance, including: (1) audio segmentation window size, (2) the type and quality of audio data, and (3) speaker overlaps. here, the best results were obtained with a model that was trained exclusively on the same type of data as the test audio files (sociolinguistic interviews). the addition of higher quality training data from the audiobook recording did not improve the model performance on the specific test data in this study. ## limitations and future research not only does the language context pose a challenge itself, but the type of training data used in this study further tests the performance of both whisper and elpis. our data consists of field recordings of sociolinguistic interviews. this introduces both environmental noise and speaker overlap into the data. other work using whisper on higher resource languages has shown better performance @xcite . crucially, in these studies, the test data was restricted in domain to elicited speech or short readings. concerning elpis, data sets containing multiple speakers in one training file are not recommended @xcite . we were also unable to account for the effects of code-switching in our data, which is likely to have impacted performance. annotating the data to identify specific segments that include code-switching would be time-consuming, especially for closely related varieties such as the ones here. research into utilizing whisper on code-switching between french and kréyòl gwadloupéyen shows similar results to those reported in this paper @xcite . nevertheless, the realities of language docu-mentation mean that data collection cannot always proceed in a way that facilitates asr model training. more work is needed to better understand how different asr systems such as whisper and elpis respond to less than ideal training data. also left for future work is a formal comparison of the time required for an asr-aided workflow vs. manual transcription of our data. other researchers have reported similar times for manual transcription vs. correcting an asr transcription @xcite . another study concludes that asr output can be useful for transcription only if the wer is less than 30%, which is considerably lower than the mean wers reported here @xcite . ## conclusion čakavian represents a low-resource context that challenges conventional asr. there exist no pretrained models for use, local varieties differ substantially from one another, and speakers employ frequent code-switching to standard croatian. to lessen transcription time, linguists are faced with modeling the data from scratch or reaching for a related language model to adapt. our results show that model adaptation is the best practice for the automatic transcription of čakavian. the collection of clean, high quality training data that better conforms to the design specifications for a tool such as elpis may allow for the creation of models that provide usable automatic transcriptions, based on a small manually transcribed dataset. however, without such training data, systems like whisper offer better performance.