End-to-End Automatic Speech Recognition: Its Impact on the Workflowin Documenting Yoloxóchitl M ixtec (2021)

Abstract: This paper describes three open access Yoloxóchitl Mixtec corpora and presents the results and implications of end-to-end automatic speech recognition for endangered language documentation. Two issues are addressed. First, the advantage for ASR accuracy of targeting informational (BPE) units in addition to, or in substitution of, linguistic units (word, morpheme, morae) and then using ROVER for system combination. BPE units consistently outperform linguistic units although the best results are obtained by system combination of different BPE targets. Second, a case is made that for endangered language documentation, ASR contributions should be evaluated according to extrinsic criteria (e.g., positive impact on downstream tasks) and not simply intrinsic metrics (e.g., CER and WER). The extrinsic metric chosen is the level of reduction in the human effort needed to produce high-quality transcriptions for permanent archiving.

## introduction: endangered language documentation history and context endangered language (el) documentation emerged as a field of linguistic activity in the 1990s, as reflected in several seminal moments. in 1991 the linguistic society of america held a symposium entitled "endangered languages and their preservation"; in 1992 @xcite published a seminal article on endangered languages in language, the lsa's flagship journal. @xcite argued for the development of documentary linguistics as an endeavor separate from and complementary to descriptive linguistics. by the early years of the present millennium, infrastructure efforts were being developed: metadata standards and best practices for archiving @xcite ; tools for lexicography and corpus developments such as @xcite , and @xcite , and financial support for endangered language documentation (the volkswagen foundation, the nsf documenting endangered language program, and the soas endangered language documentation programme). recent retrospectives on the impact of @xcite and himmelmann (1998) have been published by @xcite and @xcite . within the last decade, the national science foundation supported a series of three workshops, under the acronym aardvarc (automatically annotated repository of digital audio and video resources community) to bring together field linguists working on endangered languages and computational linguists working on automatic annotation-particularly automatic speech recognition (asr)-to address the impact of what has been called the "transcription bottleneck" @xcite . interest in applying machine learning to endangered language documentation is also manifested in four biennial workshops on this topic, the first in 2014 @xcite . finally, articles directly referencing asr of endangered languages have become increasingly common over the last five years @xcite @xcite @xcite @xcite @xcite @xcite . this article continues work on yoloxóchitl mixtec asr @xcite . the most recent efforts (2020 and 2021) have adopted the espnet toolkit for end-to-end automatic speech recognition (e2e asr). this approach has proven to be very efficient in terms of time needed to develop the asr recipe @xcite and in yielding asr hypotheses of an accuracy capable of significantly reducing the extent of human effort needed to finalize accurate transcribed audio for permanent archiving as here demonstrated. section 2 discusses the yoloxóchitl mixtec corpora, and section 3 explores the general goals of el documentation. section 4 reviews the e2e asr and corresponding results using espnet. the conclusion is offered in section 5. ## end-to-end asr experiments 4.1 experiment settings recently, e2e asr has reached comparable or better performances than conventional hidden-markov-model-based asr @xcite @xcite @xcite . in practice, e2e asr systems are less affected by linguistic constraints and are generally easier to train. the benefits of such systems are reflected in the recent trends of using end-to-end asr for el documentation @xcite @xcite @xcite . in developing e2e asr recipes for ymc, we have adopted transformer and conformerbased encoder-decoder networks with hybrid ctc/attention training @xcite . we used the ymc-exp (trainsplit) for training and other ymc corpora for evaluation. the hyper-parameters for the training and decoding follow @xcite . seven systems with different modeling units are examined in the experiments. four systems employ the byte-pair encoding (bpe) method trained from unigram language models (kudo and richardson, 2018), with transcription alphabets limited to the 150, 500, 1000, and 1500 most frequent byte-pairs in the training set. the other three asr systems adopt linguistic units, including word, morpheme, and mora. the ym word is defined as a stem with all prefixes (such as completetive ni 1 -, causative sa 4 -, and iterative nda 3 -) separated from the stem by a hyphen; and all enclitics (particularly person markers for subjects, objects, and possessors, such @xmath0, 1sg; =un 4 , 2sg; =an 4 , 3sgfem; =o 4 , 1plincl; as well @xmath1, augmentive). many vowel-initial enclitics have alternative vowels, and many encl- @xcite and as a result wrote the 1plinclusive as an independent pronoun and not an enclitic. we adopt recognizer output voting error reduction (rover) for the hypotheses combination @xcite . three combinations have been evaluated: (1) rover among only linguistic units (i.e., morae, morpheme, and word), (2) rover among only sub-word units (in this case bpe); and (3) rover combination utilizing all seven systems. ## conclusion a fundamental element of endangered language documentation is the creation of an extensive corpus of audio recordings accompanied by timecoded annotations in interlinear format. in the best of cases, such annotations include an accurate transcription aligned with morphological segmentation, glossing, and free translations. the degree to which such corpus creation is facilitated is the extrinsic metric by which asr contributions to el documentation should be considered. the project here discussed suggests a path to creating such corpora using end-to-end asr technology to build up the resources (30-50 hours) necessary to train an asr system with perhaps a 6-10 percent cer. once this threshold is reached, it is unlikely that further improvement will significantly reduce the human effort needed to check the asr output for accuracy. indeed, even if there are no "errors" in the asr output, confirmation of this through careful revision of the recording of the transcription would probably still take 3-4 hours. the effort reduction of 75 percent documented here for ymc is, therefore, approaching what may be considered the minimum amount of time to proofread transcription of natural speech in an endangered language. this project has also demonstrated the advantage of using a practical orthography that separates affixes and clitics. in a relatively isolating language such as ym, such a system is not difficult for native speakers to write nor for asr systems to learn. it has the advantage of creating a workflow in which parsed text is the direct output of e2e asr. the error rate evaluations across the spectrum of corpora and cer/wer also demonstrate the advantage of using subword units such as bpe and subsequent processing by rover for system combination (see above and a final question concerns additional steps once cer is reduced to 6-8 percent, and additional improvements to asr would not significantly affect the human effort needed to produce a high-quality time-coded transcription and segmentation. four topics are suggested: (1) address issues of noise, overlapping speech, and other challenging recording situations; (2) focus on transfer learning to related languages; (3) explore the impact of "colonialization" by a dominant language; and (4) focus additional asr-supported corpus development on producing material for documentation of endangered cultural knowledge, a facet of documentation that is often absent from endangered language documentation projects.