H ighland P uebla N ahuatl Speech Translation Corpus for Endangered Language Documentation (2021)

Abstract: Documentation of endangered languages (ELs) has become increasingly urgent as thousands of languages are on the verge of disappearing by the end of the 21st century. One challenging aspect of documentation is to develop machine learning tools to automate the processing of EL audio via automatic speech recognition (ASR), machine translation (MT), or speech translation (ST). This paper presents an open-access speech translation corpus of Highland Puebla Nahuatl (glottocode high1278), an EL spoken in central Mexico. It then addresses machine learning contributions to endangered language documentation and argues for the importance of speech translation as a key element in the documentation process. In our experiments, we observed that state-of-the-art end-to-end ST models could outperform a cascaded ST (ASR > MT) pipeline when translating endangered language documentation materials.

## introduction due to the need for global communication, computational technologies such as automatic speech recognition (asr), machine translation (mt: textto-text), and speech translation (st: speech-to-text) have focused their efforts on languages spoken by major population groups @xcite . many other languages that are spoken today will probably disappear by the end of the 21st century @xcite . for this reason, until very recently they have not been targeted for machine learning technologies. this is changing, however, as increasing attention has been paid to language loss and the need for preservation and, in best-case scenarios, revitalization of these languages. this paper presents an open-access speech translation corpus from highland puebla nahuatl to spanish and discusses our initial effort on st over the corresponding corpus. the following of this paper is organized as follows: in section 2, we discuss the benefits of speech translation for el documentation and pioneer-suggest it as the first step in the documentation process. in section 3, we compare the strategies (i.e., cascaded model and end-to-end models) that can be used to automate st for els. in section 4 we introduce the highland puebla nahuatl-to-spanish corpus. initial experimental efforts in building st models are elaborated in section 5. the conclusion is presented in in section 6. ## benefits of speech-to-text translation as a first step in language documentation the present article suggests that speech translation (st) could be a viable and valuable tool for el documentation efforts for three reasons @xcite . first, the transcription of native language recordings may become particularly problematic and time-consuming (the "transcription bottleneck") when the remaining speakers are elderly, and the younger generation has at best a passive knowledge of the language, a common situation of els. second, in many cases st may be more accurate than mt for target language translation. finally, many el documentation projects suffer from a lack of human resources with the skills and time to transcribe and analyze recordings (for similar points about a "translation before transcription workflow", see bird, 2020, section 2.2.2). by beginning with st, semi-and passive speakers can better contribute to el documentation of their native languages with a level of effort far lower than needed for transcription and analysis. bilingual native speakers or researchers with incomplete knowledge of the source language structure can quickly produce highly informative free translations even if the original text is never, or only much later, segmented and glossed. a free translation in audio and subsequent capture by typing or using asr systems for the major target l2 lan-guage (that are more accurate for major as opposed to minor and endangered languages) may take 4-5 hours of effort per hour of audio, whereas transcription (without analysis) may take 30-100 hours for the same unit. starting with free translation, then, increases the pool of potential native speaker participants and quickly adds value to an audio corpus that may languish if the first step is always fixed as transcription and segmentation (morphological parsing and glossing). in general, el documentation proceeds in a fairly set sequence: (1) record; (2) transcribe in time-coded format; (3a) analyze by parsing (morphological segmentation) and glossing; and (3b) freely translate into a dominant, often colonial, language. it may be that some projects prioritize free translation (3b) over morphological segmentation and glossing. given that each procedure adds a certain, often significant, amount of time to the processing pipeline, there is an increasing scarcity of resources as one proceeds from ( 1 ) to (3a/b). if the standard sequence is followed, there are invariably more recordings than transcriptions, more transcriptions than analyses, and (if the sequence is 3a > 3b) more analyses than free translations or (if the sequence is 3b > 3a) more free translations than analyses (see @xcite . the argument presented here is that the easiest data to obtain are the recordings followed by free translations into a major language. it may be beneficial to reorder the workflow so that an st corpus, i.e., free translation of the recording, is prioritized. only later would transcription and analysis (morphological segmentation and glossing) be inserted into the pipeline. to facilitate computational support for speech-to-text production, we would recommend a targeted number of recordings (e.g., 50 hours), followed by division into utterances with time stamps and free translation of the utterances into a major language. this corpus (or perhaps one even larger) would be used to train an end-to-end neural network in speech-to-text production. the trained st system would then be used to process additional recordings, thus generating a very extensive freely translated corpus. our hope would be that instead of basing asr on an acoustic signal alone, using two coupled inputs-the speech signal and the free translation-might well lower asr error rates from those obtained from the speech signal alone. the extent of improved accuracy is at this point simply a hypothesis. it would have to be empirically researched, something we hope to do in the near future (see anastasopoulos, 2019, chap. 4). in this scenario for el documentation, transcription and analysis proceed forward, but only after an extensive st training/validation/test corpus has been developed. the resultant st system would then be used to freely translate additional recordings as they are made. speech translation (st) is very challenging, particularly for resource-scarce endangered languages. the degree of challenge might well be reduced if corpus creation focused from the beginning on translation without intermediate steps (transcription and analysis, which would take documentation in the direction of mt). moreover, translation itself is a challenging art complicated by the lexical and morphosyntactic intricacies of languages and, more often than not, the discrepancies in vision and structure between source and target language (cf. @xcite . extremely large corpora might smooth out the edges, but if free translations are created only after transcription, then the "transcription bottleneck" will also limit the availability of free translations. limited el free translation resources, in turn, creates the danger that idiosyncratic or literal translations might dominate the training set. this is another reason to position free translation directly from a recording before transcription and analysis. free translation and textual meaning: even when a transcription has been produced and then morphologically segmented and glossed, free translations are beneficial, either generated from the transcription or directly from the speech signal. for example, although multiple sense glossing (i.e., choosing from multiple senses or functions in glossing a morpheme) clarifies ambiguous meanings, it is time-consuming for a human and challenging to automate. the semantic ambiguity of single morphemes will be mitigated if not resolved, however, if accompanied by free translations. note the following interlinearization, in which, in isolation, the meaning of the gloss line is confusing. the free translations clarifies the meaning and offers a secondary sense to the verb root koto:ni. ko:koto:nis a:t komo a:mo kiowis. 0-ko:-koto:ni-s a:-t komo a:mo kiowi-s 3sgs-rdpl-to.snap-irreal.sg water-abs if not rainirreal.sg the stream will dry up into little ponds if it doesn't rain. note also that multi-word lemmas and idiomatic expressions are in many cases opaque in word-byword (or, even more challenging, morpheme-bymorpheme) glossing. again a gloss and parallel free translation preserve literal meaning while clarifying the actual meaning to target language speakers. ## experiments in this section, we present our initial effort on building an automatic st model for el documentation. following the discussion in section 3, we compare the cascaded model with end-to-end models. to construct the cascaded model, we first conduct experiments on asr and mt, respectively. next, ## conclusions in this paper, we release the highland puebla nahuatl corpus for asr, mt, and st tasks. the corpus, related baseline models, and training recipes are open source under the cc by-nc-nd 3.0 license. we expect the corpus to facilitate all three tasks for el documentation. we also discuss and present three specific reasons for prioritizing st as an initial step in the endangered language documentation sequence after the recording has taken place. finally, we explore different technologies for st of highland puebla nahuatl and compare these to results obtained by processing through the cascaded st pipeline. as discussed in section 2, we suggest that prioritizing free translation as a first, not final, step in documentation should be considered as: (1) it can rapidly make a corpus valuable to potential users even if transcription, morphlogical segmentation, and morpheme glossing is incomplete; (2) it enables semi-, passive and heritage speakers to participate in documentation of their languages; (3) it provides an alternative process for asr in which the asr target is not a transcription but a translation into a western language; and (4) it creates a scenario in which the acoustic signal and free translation may be coupled as inputs into an end-toend asr system. therefore, our future works will focus on how the human effort could be reduced via st models and on how to incorporate st to improve the asr performances.