Enhancing Documentation of H upa with Automatic Speech Recognition (2022)

Abstract: This study investigates applications of automatic speech recognition (ASR) techniques to Hupa, a critically endangered Native American language from the Dene (Athabaskan) language family. Using around 9h12m of spoken data produced by one elder who is a first-language Hupa speaker, we experimented with different evaluation schemes and training settings. On average a fully connected deep neural network reached a word error rate of 35.26%. Our overall results illustrate the utility of ASR for making Hupa language documentation more accessible and usable. In addition, we found that when training acoustic models, using recordings with transcripts that were not carefully verified did not necessarily have a negative effect on model performance. This shows promise for speech corpora of indigenous languages that commonly include transcriptions produced by second-language speakers or linguists who have advanced knowledge in the language of interest.

## introduction the documentation of endangered and other lessstudied languages typically involves the creation of high-quality audio and video recordings representing a variety of speech genres, with the long-term goal of generating general-purpose linguistic data that can be used by diverse audiences for different research and applied purposes @xcite . with the advent of cheap, highly portable digital recording and storage technologies since the early 2000s, it is not uncommon for fieldwork projects to generate hundreds of hours of multimedia recordings. while these collections of recordings are becoming increasingly accessible via web-based portals, in the sense that they can be downloaded, locating information of interest within them correctly and efficiently is another matter entirely. coarsegrained catalog metadata describing the content of the recordings can provide users with some shallow guidance, but the identification of more specific information requires enormous investments of time and effort. accordingly, it becomes essential to have adequate transcriptions of recordings for users to find the information they are interested in. transcribing recordings, however, is also an extremely time-consuming endeavor, leading to what is sometimes called the "transcription bottleneck" @xcite @xcite , which refers to the situation where the language data is mostly in the form of (archival) recordings, and transcriptions of the data are not yet available. hupa (iso 639-3 code: hup; glottolog code: hupa1240), a critically endangered native american language of northwestern california, provides a case in point. since the early 2000s, mrs. verdena parker, an elder from the hoopa valley tribe, has generously shared her knowledge of the language with other community members and academic researchers. recordings produced by and with mrs. parker include several hours of monolingual hupa narratives and other texts, as well as over 800 hours of linguistic interviews that are a mixture of hupa and english as the elicitation metalanguage. 1 the sheer quantity of these hupa recordings makes their transcription challenging, a situation that is exacerbated by other factors. first, the people who are considered first-language speakers of hupa are older and tend not to be literate in the language. therefore the pool of potential transcribers is limited to second-language speakers and linguists with advanced research knowledge. second, while literacy is used as a tool for some pedagogical purposes in the contemporary hupa community and there is a reasonably well-established practical orthography, many of the classes for learning hupa focus more on developing oral proficiency rather than on literacy skills per se. this means many of the younger people who have become secondlanguage speakers of the language may not feel confident in their ability to produce accurate transcriptions of connected discourse. in this work, we apply automatic speech recognition (asr) technology to help address the transcription bottleneck for hupa. in particular, we hope to develop effective techniques that would lend themselves to transcribing spoken hupa. at this stage of the research, we are focusing primarily on monolingual narratives and other texts since these have the highest density of linguistic data and thus more value for research and language documentation. ## related work while research on asr for endangered language documentation is still relatively rare, recently there has been growing efforts trying to mitigate this gap @xcite . @xcite adopted end-to-end systems for yoloxóchitl mixtec, an endangered mixtecan language. using encoder-decoder architectures, they achieved the best word error rate (wer) (∼16%) for over 55h of conversational speech from more than twenty speakers. gupta and boulianne (2020) applied neural asr models for cree, an indigenous language in canada. their data consists of 4h30m story retelling or reading from six speakers. utilizing data from high-resource languages, zahrer et al. (2020) performed cross-linguistic learning of phoneme recognition for the muyu language. in a study of asr for two tonal languages, yongning na and eastern chatino, @xcite proposed a neural architecture to jointly predict phonemes and tones without needing timealigned transcripts and pronunciation dictionary. asr technologies have also been developed for some dene languages @xcite , though in a limited way. for instance, speech recognition tools were incorporated into the rosetta stone language learning software for diné bizaad (navajo). 2 the persephone asr software @xcite was combined in elan @xcite for tsuut'ina. ## results the average wer results for the verified data given each training setting and evaluation scheme are presented in when combining all data from the verified set and the coarse set together, we reached a wer of 35.26% evaluated with random splits, which is comparable to the results of random splits for each data set separately. ## discussion & ongoing work leveraging asr technologies, we investigate the possibility and effectiveness of automatically transcribing fieldwork recordings for hupa. through experimentation with different evaluation schemes and training settings, the acoustic models demonstrate reasonable wer results, showing promise for applying spoken language technology to document hupa. interestingly, training asr models using recordings with transcripts that were not carefully verified did not negatively impact the performance, which bodes well for speech corpora of indigenous languages that include transcriptions produced by second-language speakers or linguists. in ongoing work, we are extending our efforts in several directions. first, the transcripts of the coarse data are being manually checked periodically to improve transcription and gloss alignment quality. second, as we are still in the preliminary stage of performing asr for hupa, the current study only used the dnn architecture from kaldi. we plan to explore other more recent neural approaches @xcite that have been found to be effective with limited amount of audio data @xcite ; then apply the trained models to recordings that have not yet been transcribed in an iterative fashion to better combine asr with documentation of hupa. even a wer as high as ∼ 35.26% is expected to yield significant savings in the time required to make transcribed texts available. third, thus far our acoustic models are decoded with language models at the word level. however, given the complex morphological features of hupa @xcite , to reduce outof-vocabulary rate in future experiments, we are working towards combining morphological segmentation or subword unit models @xcite into building asr systems. lastly, with better performing acoustic models and more transcriptions, we aim to develop a workflow to adapt these transcribed materials into pedagogically-oriented resources for use by members of the community.