Looking within the self: Investigating the Impact of Data Augmentation with Self-training on Automatic Speech Recognition for H upa (2024)

Abstract: We investigate the performance of state-of-the-art neural ASR systems in transcribing audio recordings for Hupa, a critically endangered language of the Hoopa Valley Tribe. We also explore the impact on ASR performance when augmenting a small dataset of gold-standard high-quality transcriptions with a) a larger dataset with transcriptions of lower quality, and b) model-generated transcriptions in a self-training approach. An evaluation of both data augmentation approaches shows that the self-training approach is competitive, producing better WER scores than models trained with no additional data and not lagging far behind models trained with additional lower quality manual transcriptions instead: the deterioration in WER score is just 4.85 points when all the additional data is used in experiments with the best performing system, Wav2Vec. These findings have encouraging implications on the use of ASR systems for transcription and language documentation efforts in the Hupa language.

## introduction automatic speech recognition (asr) can assist with the manual process of transcribing audio recordings in low-resource and endangered languages, thereby facilitating language documentation efforts in these languages. with neural networks now dominating research in asr @xcite @xcite , and with related efforts to build and release open-source neural-network based asr frameworks @xcite @xcite , the possibilities for research on asr for endangered languages have greatly increased; a researcher can now leverage one of the open-source asr toolkits and apply it to their language of interest. cross-lingual speech representations currently leveraged by state-of-the-art neural asr systems @xcite also provide opportunities for knowledge transfer to endangered languages; commonalities across different speech representations can be leveraged to improve asr performance. the hupa language from the dene/athabaskan language family is one such language that stands to benefit from these advances in asr. hupa is the ancestral language of the hoopa valley tribe residing in northern california. it is critically endangered with only a handful of first-language (l1) speakers and a number of second-language learners. since the 1970s, the hupa speech community has been actively engaged in documentation and reclamation work to preserve their language. asr systems can be especially beneficial for hupa, given that there may be low literacy levels among hupa speakers and learners of the language who focus instead on oral proficiency; these low literacy levels may in turn hinder efforts to transcribe audio recordings. however, the development of an asr system for hupa using supervised learning approaches faces a chicken-and-egg problem: high quality transcriptions are necessary to train a performant asr system which can be leveraged in a manual transcription process to produce high quality transcriptions. given the challenges with producing additional manual annotations, data augmentation approaches must instead be relied upon to generate the necessary data to train an asr system. in this study, we explore the efficacy of different asr systems for hupa, coupled with selftraining, a data augmentation method that is favored in the literature due to its simplicity and elegance @xcite . the general idea is to apply a trained model to unlabeled data, then combine the automatically annotated data with existing gold-standard training data to build a new model, to see whether the addition of model-generated annotations is helpful towards model performance. with this approach, here we seek to investigate if model-generated au-dio transcripts can be leveraged to improve acoustic model performance on gold-standard data with high-quality manual transcriptions. in addition, we compare the self-training approach to an alternative method, where we swap out the additional machine-produced transcripts in the training data with human-annotated data that has overall lower transcription quality (section 3). in addition, others investigate augmentation methods applied on the acoustic signals to improve asr performance for endangered languages. these include, but are not limited to, semisupervised training and vocal tract length perturbation @xcite , elastic spectral distortion methods @xcite , creation of synthetic data using voice transformation and signal distortion @xcite and transfer learning with data augmentation @xcite . self-training some recent studies have begun to apply self-training, or "pseudo-labeling", for asr, mostly focusing on english @xcite @xcite . a few multilingual studies exist @xcite , including investigations of cross-linguistic lowresource settings @xcite . for endangered languages, @xcite apply self-training to four minority languages, gronings, west-frisian, besemah, and nasal, each with 4h of acoustic training data to start with. in comparison to previous approaches, our work uses the wav2vec 2.0 framework @xcite , and generates transcripts from audio data in a self-training setting while leaving the audio intact. self-training has not been widely applied to endangered languages with the exception of @xcite , who do not include hupa in their study; the amount of gold-standard high quality acoustic training data that we start out with is also quite small in comparison with their work (more details in section 3). ## the hupa asr dataset the audio data for hupa is a result of continuous linguistic fieldwork since 2005. the spoken records are provided by mrs. verdena parker, an l1 speaker of the language. the audio content is composed of different genres including descriptions of historical tribal events and stories and tales narrated by mrs. parker. the transcriptions of the audio files have been carried out by several linguistics researchers over the years with consultation from mrs. parker. each transcript follows the practical orthography developed in @xcite , and is time aligned with annotation tools such as elan @xcite . transcripts go through stages of manual verification to different extents, which are necessary since the recordings come from multiple fieldwork sessions across different years and are transcribed by different researchers. some transcripts are checked more thoroughly than others, with more checks resulting in better transcription quality. based solely on transcription quality, we divide the audio data and their corresponding transcripts into two datasets: the "fine" and the "coarse" datasets. the fine data has approximately 1h35m of audio with thoroughly checked transcriptions, and the coarse data has around 7h37m of audio with comparatively lower transcription quality. ## results it is clear from comparing the two model architectures, wav2vec2 outperforms kaldi across all experiments in terms of absolute wer. both architectures are able to utilize both manuallytranscribed and model-generated data to achieve improvements in wer, though the gains are slightly bigger with the former. interestingly, kaldi seems to better utilize the model-generated transcripts; looking at the best score with selftraining normalized by the baseline, kaldi shows an improvement of 13.91 percentage points over the baseline compared to 12.44 points for wav2vec2. these numbers suggest that wav2vec2 is possibly more sensitive to the noise present in model-generated transcripts. while a self-training setup with model-generated transcripts under-performs versus training with additional coarse transcripts of the same size ( lastly, the impact of using additional data from self-training versus manual annotation can be viewed from the perspective of an investigation of word types present in the transcriptions of each approach. specifically, we look at the number of new word types introduced by each approach that are not present in the fine dataset; the coarse dataset from the 5x partition contains 5,521 new types not present in the fine dataset, versus 8,487 in the model-generated transcriptions. given the similar number of tokens between the two, the modelgenerated transcripts have a higher type-token percentage of 17.71 when considering only new types (cf. 11.42 in the coarse transcripts), which seems to correlate with higher wer scores. however, of the 5,521 new types in the coarse transcripts, 2,516 (46%) are also found in the model-generated transcripts. this has implications for new vocabulary discovery in language documentation efforts for hupa, as up to 46% of new words in the coarse dataset can be discovered through asr transcriptions instead of solely through manual effort. ## conclusion and future work we find that a self-training approach for hupa asr is able to produce transcriptions of better quality than one using no additional training data, as seen in the 3.67 point improvement in wer score; moreover, it does not fall far behind when compared to the best-performing setup of using all the additional human-transcribed coarse data (a 4.85 wer difference). moreover, the availability of manually verified low-quality transcripts in the best performing setup should not be taken for granted; it is not uncommon, in the cases of indigenous and endangered languages, for the audio recordings to be sourced from just one speaker (see also @xcite ), and the resources needed to produce transcriptions from the audio may be very limited. with that in mind, we believe self-training to be a useful data augmentation method, at least in the initial stages of developing asr systems for endangered languages when there is very little data available. an important goal of developing asr systems for endangered languages is to automate, fully or partially, the transcription of new fieldwork recordings; the automatic transcriptions can be manually corrected by speakers of the language, potentially removing the need to transcribe from scratch @xcite . it would be interesting to study whether an asr system trained on additional manually produced transcripts of any quality would be more beneficial to the transcription process than a system trained on additional model-generated transcripts using self-training; or in other words, can transcribers tolerate a 4.85 point degradation in wer score by using model-generated transcripts to train an asr system, in exchange for not needing to manually produce transcriptions to improve that system? additionally, would tools such as elpis @xcite that take advantage of speech recognition technologies in their language documentation transcription workflows benefit from the integration of self-training or other data augmentation methods into existing pipelines? we leave these possibilities for future work.