Multilingual Models for ASR in Chibchan Languages (2024)

Abstract: We present experiments on Automatic Speech Recognition (ASR) for Bribri and Cabécar, two languages from the Chibchan family. We fine-tune four ASR algorithms (Wav2Vec2, Whisper, MMS & WavLM) to create monolingual models, with the Wav2Vec2 model demonstrating the best performance. We then proceed to use Wav2Vec2 for (1) experiments on training joint and transfer learning models for both languages, and (2) an analysis of the errors, with a focus on the transcription of tone. Results show effective transfer learning for both Bribri and Cabécar, but especially for Bribri. A post-processing spell checking step further reduced character and word error rates. As for the errors, tone is where the Bribri models make the most errors, whereas the simpler tonal system of Cabécar is better transcribed by the model. Our work contributes to developing better ASR technology, an important tool that could facilitate transcription, one of the major bottlenecks in language documentation efforts. Our work also assesses how existing pre-trained models and algorithms perform for genuine extremely low resource-languages.

## introduction this paper presents experiments on automatic speech recognition (asr) for bribri and cabécar, two languages belonging to the chibchan family and spoken in costa rica. the data available for modeling is extremely limited, with only 143 minutes for bribri and 54 minutes for cabécar. in such a context, conventional natural language processing (nlp) tools relying on deep neural networks trained from scratch are not an option, but leveraging pre-trained models offers a solution. our first steps will be to compare the performance of different algorithms when tackling these extremely under-resourced languages. we used four asr algorithms to finetune their base model and thereby train monolingual models for bribri and cabécar. we then used the best performing algorithm, wav2vec2, to train two additional types of models: i) a joint model simultaneously trained on both languages, and ii) a transfer learning model initially trained on one language and subsequently on the other. then, we evaluated all models on each language independently. next, we applied a unigram spell checker to correct the final output. our results indicate that transfer learning is effective for both bribri and cabécar, with better performance in transferring from cabécar to bribri. in contrast, the joint models did not surpass the mono-lingual or transfer models. the post-processing spell-checking contributed to reducing the word error rate (wer) for cabécar. finally, we performed an error analysis of the monolingual and transfer models to test our intuition that linguistic tone presents more issues during transcription than other aspects of chibchan phonology. asr is potentially a valuable tool for field linguists working in language documentation and revitalization, given that their primary data source comprises recorded interviews and elicitation sessions with l1 speakers. asr is also promising for developing tools directly accessible to the members of these indigenous communities. as speech serves as a more immediate form of communication than writing, asr-based tools could be used for capturing diverse communication interactions between speakers, encouraging the use of the language. the challenges inherent in working with these languages are substantial, distinguishing it from conventional nlp tasks. the audio files can present less-than-ideal recording conditions, with background noise, and the speakers may be elderly community members, potentially with communication difficulties. the impact of tokenization tools remains unclear, especially considering that the writing systems may not be stable. on the other hand, engaging with these linguistically diverse languages not only stress-tests our models but also encourages inclusivity in research. the complexities involved in addressing their unique linguistic features contribute to a broader understanding of the capabilities and limitations of asr in diverse linguistic contexts. ## experiments in this section we will describe the data sources, the algorithms used for asr training, and the methods for evaluating the experiments. ## results in this section we will present four results: (i) the quantitative measurements of the character and word error rates for models trained on bribri and cabécar data, using different algorithms, (ii) the differences in performance between the monolingual, joint and transfer models, (iii) a qualitative analysis of the differences in the transcriptions from the different models, and (iv) an error analysis of different components of bribri phonology and how those are transcribed by the best performing models. ## discussion regarding the type of learning, transfer learning proved effective for both bribri and cabécar, with superior performance for bribri. the joint models did not surpass the effectiveness of the monolingual ones, but at least for bribri, they closely followed the performance of the transfer-based models. regarding the post-processing, spell-checking only contributed to reducing the wer for cabécar. contrary to findings in the multilingual nmt literature, where transfer approaches involving sequential training of languages tend to underperform compared to joint approaches where both languages are input simultaneously, our results present the opposite trend-the transfer approach yielded better results. the result that both transfer and joint models outperformed in bribri compared to cabécar strongly suggests that the "cleanliness" of the data is a very important and determinant factor. the cabécar data, characterized by fewer genres and speakers from a single geographic region, provides a more consistent signal that may result in fewer structural errors. despite having more bribri data, the model seemingly grasps cabécar better, enabling better generalization to the bribri data. in contrast to multilingual nmt techniques, our joint model does not use language tags when inputting both languages. introducing language tags would require modifications to the audio files. we posit that such modifications could enhance the model's ability to distinguish between the languages more accurately, especially given the subtle "bumps" observed in the cabécar plots which might be hinting a potential confusion. additionally, this change might improve regularization by effectively increasing the number of training examples, aligning with the patterns observed in the multilingual nmt literature @xcite . these tags will be added in future work. the correction post-processing was more effective for cabécar, suggesting once again that variation in the bribri data is high, and that the cabécar data has greater consistency. the cabécar data also has lower lexical diversity, a factor which might directly impact the spell-checker's performance. critically, these improvements concerned the wer, but the cer did not show much gain. one plausible explanation is that the acoustic model may already be saturated, meaning that it does not benefit as much from additional data. another possibility is that the pre-trained model lacks sufficient typologically relevant knowledge for the chibchan languages under examination. for instance, linguistic features like tones may be insufficiently represented. this potential deficiency serves as an example of how applying our methods in the context of languages divergent from the typical scope of languages in nlp papers allows us to assess the effectiveness of our tools. there is potential future work in expanding this dataset to perform zero-shot transcriptions of other, even lower-resourced chibchan languages such as malecu and ngäbere, and hopefully including them into a large pan-chibchan multilingual model in the future. such a model would enhance the transcription accuracy of the existing languages. ## conclusions we presented experiments on multilingual asr learning for bribri and cabécar, two extremely lowresource languages spoken in costa rica. we have shown that cross-lingual learning does take place between languages, specifically from cabécar to bribri. we have also shown that the quality of the data is highly significant in resource-constrained settings where each training data point holds considerable weight. our results are promising but we have not explored the full potential of this technique. for example, further data processing could involve the inclusion of language tags and their corresponding audio equivalents. another possibility is is selecting a subset of the bribri data that aligns more closely with the cabécar data in terms of domain. additionally, future research could consider zero-shot evaluation on languages within the same linguistic family.