Phoneme transcription of endangered languages: an evaluation of recent ASR architectures in the single speaker scenario (2022)

Abstract: Transcription is often reported as the bottleneck in endangered language documentation, requiring large efforts from scarce speakers and transcribers. In general, automatic speech recognition (ASR) can be accurate enough to accelerate transcription only if trained on large amounts of transcribed data. However, when a single speaker is involved, several studies have reported encouraging results for phonetic transcription even with small amounts of training. Here we expand this body of work on speaker-dependent transcription by comparing four ASR approaches, notably recent transformer and pretrained multilingual models, on a common dataset of 11 languages. To automate data preparation, training and evaluation steps, we also developed a phoneme recognition setup which handles morphologically complex languages and writing systems for which no pronunciation dictionary exists. We find that fine-tuning a multilingual pretrained model yields an average phoneme error rate (PER) of 15% for 6 languages with 99 minutes or less of transcribed data for training. For the 5 languages with between 100 and 192 minutes of training, we achieved a PER of 8.4% or less. These results on a number of varied languages suggest that ASR can now significantly reduce transcription efforts in the speaker-dependent situation common in endangered language work.

## introduction recent progress in automatic speech recognition (asr) was made by training neural networks on increasingly large amounts of annotated data. to significantly reduce the efforts needed to transcribe endangered languages, asr must reach sufficient accuracy when trained on relatively much smaller amounts of transcribed data. already several research efforts have been dedicated specifically to asr for low-resource languages, such as the iarpa babel program foot_0 and the nist openasr challenge foot_1 . however, creating an asr system for a task like speaker-independent phonetic transcription is still difficult and requires amounts of transcription that are very large in the context of endangered languages. for example, @xcite recently concluded that at least 50 hours of training data are needed for this task, comparing espnet and hmm-based models on two languages. in language documentation, field recordings are seldom made with a large number of speakers, but rather with a few speakers and for long durations @xcite . in these conditions, small amounts of transcribed data from a single speaker might be enough to train a phoneme recognizer with sufficient accuracy to automatically transcribe the remaining recordings from the same speaker. concentrating on the single speaker scenario, @xcite evaluated a ctc-based lstm model on na and chatino, and showed encouraging results for automated phoneme transcription as well as the effectiveness of this approach for linguistic work on endangered languages; they also created the open-source phonemic transcription tool persephone. @xcite compared persephone performance on several endangered languages, focussing on data preprocessing concerns. gupta and boulianne (2020) compared end-to-end persephone and wav2letter++ with an hmm-blstm hybrid for single speaker phoneme transcription, but using only one language, cree. more recently, @xcite evaluated es-pnet on na, chatino and japhug and integrated it into elpis to create a user friendly docker container. although these previous studies obtained promising results, they report on different systems and languages, making them difficult to compare. in addition, none has yet evaluated fine-tuning recent large models pretrained on many languages, for example xlsr @xcite foot_2 , which are particularly well suited for low-resource languages. we think it fair to include such models, as we aim at a practical solution for the transcription problem at hand, regardless of the underlying approach. in this paper we extend the body of work on single speaker phonetic transcription for endangered or low-resource languages while introducing distinctive contributions. for a meaningful comparison, we evaluate 4 systems with different modeling approaches across a common set of 7 languages, and 3 of those systems across 11 languages, while previous work was limited to either a single system on many languages, or many systems on a single language. in addition to persephone and hmm-gmm models, we compare two recent architectures that have never been evaluated for single-speaker phoneme recognition: a conformer model with a lf-mmi criterion, and a large pretrained multilingual model that we fine-tune for this task. we more firmly establish feasibility of accurate phonemic transcription with 3 hours or less of transcribed data by reporting on 4 new languages, including cree and highly polysynthetic inuktitut, in addition to 7 other previously studied in the literature. finally, for reproducibility we make publicly available the curated dataset of public languages and a platform-independent container which allow users to reproduce the experiments from this paper foot_3 or train their own phoneme recognizer for a new endangered language. ## datasets in this section we present the two sources of data used in the experiments. although a number of low-resource language datasets are publicly available, very few provide enough data per speaker for speaker-dependent training. for example, the maximum duration from a single speaker in babel languages is limited to 20 minutes. ## stp test bed in order to make a fair comparison, all models are evaluated through the same speech-to-phoneme recognition test bed. called stp, it automates the steps required to train a phoneme recognizer from scratch i.e., with only a small number of audio files manually transcribed using a common transcription tool such as elan. once trained, the recognizer can be applied to other audio files and yield the time-aligned phonetic transcription, in text or as elan annotations. the following sections detail the principles and design choices that were made to ensure stp could handle all the languages involved in the experiments, making it applicable to a wide range of features frequently encountered in endangered languages. ## experiments we evaluated models from four main classes: a conventional hidden markov models with gaussian mixture models (hmm-gmm), an end-to-end recurrent neural network, a convolutional/transformer-based neural network, and a large pretrained transformer neural network. we compare time required for training, hardware and software requirements, and accuracy of transcription. for a fair comparison, all models are trained and evaluated using the same stp test bed and languages. only the training pipeline needs to be run since it includes computation of phoneme error rate on the held-out part of the dataset. for a given model, the same hyperparameters were used across all languages, and are taken from the reference published paper (except where differences are noted in following sections). the test set is used only for measuring phoneme error rate and is not involved in any tuning. ## results the four architectures are compared in terms of phoneme error rate, and elapsed time for training, in in each table, languages appear in descending order of total audio duration available for training. note that the nru33 subset is used here rather than the full nru, to make it more comparable with other languages. true in the ipa column indicates that transcriptions are ipa symbols, false means that transcriptions are orthographic. phoneme error rates (per) reported are obtained using the speaker turn segmentation from the transcript. in an actual transcription pipeline, vad would be used and might introduce errors that could slightly degrade the actual per. also note that the reference is the phoneme string generated by the g2p table, so tone or stress errors are not counted if tone or stress is not represented by distinct phonemes in the table. ## conclusion fine-tuning a large pretrained multilingual model clearly outperformed the other approaches. for the 6 languages with 99 minutes or less of training data, the pretrained model was able to average a phoneme error rate of 15.3%. we obtained 8.4% or less per for the 5 languages which had between 100 and 192 minutes. at this level of performance, we expect asr to significantly reduce the effort required for transcription of endangered languages. further work is needed to explore handling of tone and stress markers, and enlarge the curated speakerdependent dataset with other publicly available languages.