Are modern neural ASR architectures robust for polysynthetic languages? (2024)

Abstract: Automatic speech recognition (ASR) technology is frequently proposed as a means of preservation and documentation of endangered languages, with promising results thus far. Among the endangered languages spoken today, a significant number exhibit complex morphology. The models employed in contemporary language documentation pipelines that utilize ASR, however, are predominantly based on isolating or inflectional languages, often from the Indo-European family. This raises a critical concern: building models exclusively on such languages may introduce a bias, resulting in better performance with simpler morphological structures. In this paper, we investigate the performance of modern ASR architectures on morphologically complex languages. Results indicate that modern ASR architectures appear less robust in managing high OOV rates for morphologically complex languages in terms of word error rate, while character error rates are consistently higher for isolating languages.

## introduction traditional morphological typology @xcite recognizes a range of morphological complexity in the world's languages. at one end are isolating languages, like mandarin, where grammatical information is not encoded in affixes and words typically consist of one or more lexical morphemes. languages that do use affixes to indicate grammatical information, such as person and tense on a verb, or case and number on a noun, are typically described as either fusional -where a single morpheme encodes multiple features, as in russian, sanskrit, french -or as agglutinative -where each morpheme has a single function, as in turkish. languages with an extreme degree of morphological complexity, including phenomena like noun incorporation and "sentence words", are often described as polysynthetic. many languages indigenous to the americas, northern australia, new guinea, and siberia are polysynthetic. english, while with rich derivational morphology, is quite limited in its inflectional morphology with only a handful of regular grammatical suffixes, making it closer in many ways to an isolating language. historically, nlp has often operated under the assumption that techniques effective for english will work well for other languages. however, this is gradually changing with the emergence of highly multilingual models that account for a significant diversity in language morphology @xcite . in areas such as machine translation, text generation, and speech recognition, the broader adoption of methods like byte pair encoding @xcite has facilitated the inclusion of languages with more complex morphology. despite these advances, however, polysynthetic languages remain largely absent from large language models and automatic speech recognition systems. in this paper, we select a variety of languages with differing degrees of morphological complexity to examine the effectiveness of state-of-the-art automatic speech recognition (asr) models for transcription. we focus specifically on one key aspect: the potential for these languages to have a high number of out-of-vocabulary (oov) words due to their extensive word construction possibilities and the models' ability to handle this challenge. on one hand, these models are trained on a large number of languages, including some agglutinative ones, and they operate on subwords, which might be sufficient to manage the production of long words with multiple morphemes. on the other hand, no polysynthetic languages were included in the original pretraining sets, leaving the ability of these models to handle such complex morphology unknown. using corpora for ten languages ranging typologically from isolating to polysynthetic, we extract several features that capture aspects of morphological complexity. we then fine-tune asr models for different splits of each corpus using two stateof-the-art multilingual asr models. comparing performance for each language in terms of word error rate (wer) and character error rate (cer) across models and splits, we find that both asr architectures are less robust for polysynthetic languages and that certain measures of morphological complexity are associated with degradation of both wer and cer. ## related work much of the nlp research on polysynthetic languages has focused primarily on computational morphology with the goal of developing methods for identifying the component parts of complex word structures. early work included approaches such as diphones @xcite , bayesian methods @xcite , and transitional probabilities @xcite . hybrid methods such as adaptor grammars were also recognized as powerful tools for uncovering subword units, aiding in language documentation @xcite @xcite . recently, the trend has shifted towards combining traditional methods like finite-state transducers (fsts) with neural networks to segment words and analyze their glosses. these approaches leverage human expertise and employ technology to scale up to larger datasets @xcite @xcite . methods like bpe @xcite and sentencepiece @xcite are currently in wide use as a substitute or proxy for linguistically-informed morphological segmentation because of their efficiency, simplicity, and scalability. they have also been widely adopted into modern language processing pipelines, including contemporary text generation models (e.g. @xcite . the idea of incorporating asr into language documentation pipelines started to be explored with hmm-based architectures and has resulted in reductions in both speed of transcription and transcription error rates. @xcite rajan, 2023) but also more complex methods that rely on wfsts @xcite . the expansion of transformer-based architectures has significantly enhanced the potential of asr in documentary linguistics. pretrained models like the whisper suite @xcite , wav2vec models @xcite , and mms @xcite have drastically reduced the amount of data needed to achieve reasonable results. these models have demonstrated efficiency in real-world documentary pipelines @xcite . however, their effectiveness for morphologically complex languages remains uncertain. an examination of the data used to build these models reveals that the vast majority of the included languages have relatively simple morphology, and none are polysynthetic (see ## data for our study, we will compare asr performances on three mostly isolating languages: namakura, east uvean, and bambara, and seven languages with different levels of morphological complexity: kanakanavu, rukai, hupa, enenlhet, kunwok, plains cree, and st lawrence yupik. namakura (iso-nmk) and east uvean (iso-wls) are austronesian languages spoken respectively on wallis island and vanuatu in the pacific ocean. they are both isolating languages with simple morphology. both corpora are spontaneous fieldwork recordings extracted from the pangloss database foot_0 . the total durations of the collections are 2h9m for east uvean and 2h18m for namakura. bambara (iso-bam) is one of the primary spoken languages in mali and is part of the mande language family. the language is isolating with simple morphology. the audio data consists of fieldwork recording of spontaneous speech. the total duration of the collection is 7h11m. while the corpus is not currently publicly available, it is expected to be released this year @xcite . kanakanavu (iso-xnb) and rukai (iso-dru) are members of formosan language family, the indigenous languages of taiwan. although not polysynthetic, these languages prominently feature complex morphological features such as distinct voice and case markings. the audio data consists of a mixture of fieldwork recording through the ntu corpus foot_1 (wen @xcite and pedagogical resources from epark foot_2 . both collections are part of the formosanbank project @xcite . the total duration of the collections are 4h44m for kanakanavu and 3h35m for rukai. hupa (iso-hup) is a critically endangered native american language spoken in the hoopa valley in northern california in the united states. it is part of the dene/athabaskan language family and has polysynthetic morphology. the audio data consists of narratives from one female elder native speaker of the language. the total duration of the collection is 9h12m. this corpus was shared for research purposes by a linguist who has been working with the hupa community for a decade, with consent from the elder speaker. due to ethical considerations of data sovereignty the data is not publicly available. toba-maskoy or enenlhet (iso-tmf) is an indigenous language spoken in paraguay. it is part of the maskoy language family. the language is polysynthetic but has very little documentation. the audio data consists of spontaneous stories about life and cultural topics. the total duration of the collection is 5h17m. the corpus was extracted from the archive of the indigenous languages of latin america foot_3 . bininj kunwok (iso-gup) is an aboriginal language spoken in northern australia. it is part of the macro-gunwinyguan language family and is polysynthetic. the audio data consists of guided tours by seven men and elicited sentences from three women speakers. the total duration of the collection is 1h3min. while the community has granted permission to use this data for research purposes, the corpus is not publicly available. plains cree (iso-crk) is an indigenous language spoken in the plains of western canada. it is part of the algonquian language family and is highly polysynthetic but relatively agglutinative. the au-dio consists of a reading-out-loud of the book of psalms by a fluent native speaker of plains cree. while various forms of audio for the entire bible are available online, members of our research team have previously received access to high-quality recordings of psalms for the purpose of developing a speech-synthesizer @xcite and other research, and we make use of the same audio in this work. the total duration of this selection is 2h4min. st lawrence yupik (iso-ess) is an indigenous language spoken on st lawrence island in the bering straight in alaska. it is part of the eskimo-aluit language family and is highly polysynthetic. the audio data consists of readings of the entire bible. the total duration of the collection is 33h32m. as noted above, all the languages used for this project are either extracted from open-source data collections constructed for research purposes or were made available to us through explicit agreements from the language community or associated scholars. all the languages here are indigenous and most of them are endangered. none of these languages is included in the original training set of either of the state-of-the-art asr models described later. to our knowledge, none of these languages display unusual phonetic or phonological features that could bias the experiments described below. ## results all the figures are sorted from the least to the most morphologically complex language. the language order is the following: namakura (nmk), east uvean (wls), bambara (bam), kanakanavu (xnb), rukai (dru), hupa (hup), enenlhet (tnf), kunwok (gup), plains cree (crk) and yupik (ess). the wers for all the models trained can be found in figure 3 . the performance of the models across languages cannot be directly compared due to influencing factors beyond the language's typology, such as the quantity of data and the quality of the recordings. for example, yupik (ess), whisper whisper xlsr xlsr rand max rand max correlation -0.9 -0.8 -0.9 -0.9 p-value 0.0002 0.001 0.000 0.0003 ## conclusion in this paper, we investigated how morphological complexity affects the performance of state-of-theart asr models. we focused on one key aspect: the potential for polysynthetic languages to generate a vast number of morpheme combinations, leading to a high number of unknown words during model testing. we selected a set of 10 typologically diverse languages, created datasets with varying oov rates, and attempted to correlate the degradation of wer with factors related to morphological complexity. we found that modern asr models are generally less robust for polysynthetic languages. among the various morphological factors we examined, a higher token-type ratio is associated with greater degradation of both wer and cer for morphologically complex languages. however, these models tend to produce higher cer for isolating languages. the primary aim of this study was to clarify the relationship between complex morphology and asr performance. along the way, several research questions emerged, such as the impact of elements like the diversity of data collections and the combined effect of various influences. additionally, the elements influencing the word formation capability of these models remain unclear. morphological complexity is a topic that triggers interest from a small but committed research community, but their interest is often limited to textual data. extending research contributions to speech data could lead to improved understanding of the current biases within modern asr systems and new directions in using asr for language documentation purposes.