Automated speech tools for helping communities process restricted-access corpora for language revival efforts (2022)

Abstract: Many archival recordings of speech from endangered languages remain unannotated and inaccessible to community members and language learning programs. One bottleneck is the time-intensive nature of annotation. An even narrower bottleneck occurs for recordings with access constraints, such as language that must be vetted or filtered by authorised community members before annotation can begin. We propose a privacy-preserving workflow to widen both bottlenecks for recordings where speech in the endangered language is intermixed with a more widely-used language such as English for meta-linguistic commentary and questions (e.g.What is the word for ‘tree’?). We integrate voice activity detection (VAD), spoken language identification (SLI), and automatic speech recognition (ASR) to transcribe the metalinguistic content, which an authorised person can quickly scan to triage recordings that can be annotated by people with lower levels of access. We report work-in-progress processing 136 hours archival audio containing a mix of English and Muruwari. Our collaborative work with the Muruwari custodian of the archival materials show that this workflow reduces metalanguage transcription time by 20% even given only minimal amounts of annotated training data, 10 utterances per language for SLI and for ASR at most 39 minutes, and possibly as little as 39 seconds.

## introduction in speech recorded for language documentation work, it is common to find not only the target language that is being documented but also a language of wider communication, such as english. this is especially so in early-stage fieldwork when the elicitation may centre around basic words and phrases from a standard word list (e.g. the swadesh list: @xcite . in these mixed-language recordings, utterances in the language of wider communication are largely metalinguistic questions and commentary (e.g. what is the word for 'tree'?, this word means 'soft'), which appear inter-mixed with the utterances of interest in the target language. in this paper, we propose a workflow to help process hundreds of hours of unannotated speech of this genre. we describe a use case where the language of wider communication is english (iso 639-3: eng), and the documented language is muruwari (iso 639-3: zmu), an aboriginal language traditionally spoken in north western new south wales, australia. as illustrated in figure 1 , we leverage voice activity detection (vad) to detect speech regions, then spoken language identification (sli) to distinguish between muruwari and english regions, and then automatic speech recognition (asr) to transcribe the english. the uncorrected transcriptions offer a rough but workable estimate of the contents in a given recording. ## project background muruwari is an aboriginal language traditionally spoken in north western new south wales, australia and belongs to the pama-nyungan family of australian languages @xcite . @xcite , which comprises the largest extant single work on muruwari, describes it as a relative isolate compared to the neighbouring pama-nyungan languages, yuwaaliyaay, yuwaalaraay, barranbinya, ngiyampaa (ngemba), guwamu and badjiri. james 'jimmie' barker , the last first language (l1) speaker of muruwari, produced in the early 1970s a total of 136 hours of reel-to-reel tape recordings consisting of a mix of muruwari and meta-linguistic commentary on the muruwari language in english. the now digitised recordings are held at the australian institute of aboriginal and torres strait islander studies and access to these materials depend on permission from the custodian and muruwari elder, roy barker (author rb; grandson of jimmie barker). to date, rb has manually auditioned approximately 40 of the 136 hours over the course of 4 years to determine regions of speech appropriate for general access and those requiring restricted access (e.g. for only the muruwari community, or only the barker family). at this rate of roughly 10 hours per year, the remaining 96 hours may require nearly a decade of manual review by rb. parallel to the review of the remaining recordings, a subset of the recordings that have already been cleared by rb is being used to search for excerpts that may be useful for learning materials and those that can inform the development of a standardised orthography for muruwari. to assist these ongoing initiatives, we investigated how sli and asr can be leveraged to allow for the review process and excerpt searches to be done more strategically and efficiently. ## research questions there has been growing interest in leveraging speech processing tools to assist in language documentation workflows, including the formulation of shared tasks (e.g. @xcite . 2 aimed at making unannotated fieldwork recordings more accessible, @xcite proposed a family of shared tasks, dubbed the "grandma's hatbox", which include sli and asr. in our work, we additionally leverage vad to make the system fully automatable and, to derive a rough index of the corpus, we transcribe all speech regions detected as english (in the shared task formulation, asr was intended to transcribe only the metadata preamble in the recordings). the performance of speech processing systems can be poor when there are mismatches between the speech on which they were trained and that on which they are deployed. commenting on such poor deployment-time performance of sli systems, @xcite concluded that what is necessary for real-world usage are methods for system adaptation with a few examples from the target speakers/domains. accordingly, we sought to answer the following questions: 1) how many utterances of english and muruwari are needed to adapt an off-the-shelf sli system? 2) is it possible to make use of such a system without computeintensive adaptation methods requiring a graphics processing unit (gpu)? regarding this latter question, we were inspired by a recent probing study on various speech representations showing that logistic regression classifiers performed on-par with shallow neural networks for two-way classification of speech, e.g. distinguishing between vowels and non-vowels @xcite . hence, we examined through our sli experiments whether using a logistic regression classifier suffices for the twoway classification of the speech data, i.e. distinguishing between english and muruwari. turning now to asr, the typical use case in language documentation work has been to develop asr systems to help transcribe the target language (e.g. @xcite @xcite . by contrast, our use of asr more closely aligns with recent work exploring techniques such as spoken term detec-tion to help locate utterances of interest in untranscribed speech corpora in the target languages @xcite @xcite . in this work, however, we take advantage of the mixed-language speech in the corpus, and leverage sli and asr to transcribe the english speech as a way to derive a rough index. we opted to use the robust wav2vec 2.0 model @xcite to reduce the mismatch in audio quality between the training and the deployment data (i.e. noisy archival recordings). this model is pre-trained not only on librispeech (960 hours @xcite and common-voice english (700 hours: @xcite , but also on noisy telephone-quality speech corpora (fisher, 2k hours: @xcite , and also fine-tuned on 300 hours of transcribed speech from switchboard. with our asr experiments, we sought to answer the following questions: 1) what amount of transcribed speech is sufficient to reliably achieve better than off-theshelf performance? 2) using the same amount of transcribed speech, to what extent can asr system performance be further increased when supplemented with a language model trained on external texts? ## data: the jimmie barker recordings to gather training and evaluation data for the two speech processing tasks, we obtained 6 archival recordings of jimmie barker's speech cleared by rb. for each recording, we used the off-the-shelf robust wav2vec 2.0 @xcite , foot_2 to simply transcribe all speech regions detected by the silero vad system, foot_3 and generated an .eaf file for elan. 5 using elan, three annotators (2 recordings per annotator) then erased the spurious text for the muruwari utterances (i.e. for sli, we simply used blank annotations to denote muruwari regions, given the orthography is still in development) and manually corrected the english transcriptions for asr (i.e. for sli, any non-blank region with text was considered english). while the machine-generated annotations for the training and evaluation data were human-corrected, we have yet to establish inter-annotator agreement or conduct error analyses. when correcting the english transcriptions, speech was transcribed verbatim with no punctuation except for apostrophes, i.e. including false starts (e.g. we we don't say) and hesitations (e.g. and uh it means steal). to facilitate searches, transcriptions were made in lower-case with the exception of proper nouns (e.g. uh the ngiyaamba has it uh) and words that were spelled out by jimmie (e.g. you've got b u at the end of a word). for asr training, the transcriptions were automatically converted to all upper-case to normalise the text to a 27-character vocabulary (26 upper-case letters + apostrophe) that matches vocabulary with which the wav2vec 2.0 robust model was originally trained. as we report in appendix a, not re-using the original vocabulary required significantly more fine-tuning data to achieve the same performance. based on the corrected annotations, we extracted the speech regions into individual 16-bit 16 khz .wav files and all the transcriptions for the english utterances into a single tab-delimited file. a summary of the data used in this paper is given below in ## spoken language identification we are interested in finding the minimum amount of training utterances required to obtain a performant system for same-speaker sli. as training a system with very few utterances can lead to a large variance in its performance on unseen utterances, we were particularly interested in determining the training set size at which the variance was functionally equivalent to training on all available data. ## automatic speech recognition recall that for asr, we seek to answer the following questions: 1) what amount of transcribed speech is sufficient to reliably achieve better than off-the-shelf performance for transcribing jimmie's australian english? 2) using the same amount of transcribed speech, to what extent can asr system performance be further increased when supplemented with a language model trained on external texts? in this section, we report on experiments conducted in order to answer these questions. ## timed annotation tasks in addition to helping provide estimates of the contents of recordings for review by an authorised person, another purpose of this workflow is to help reduce the time required to annotate speech in such a way that excerpts from cleared recordings can be easily extracted for use in relevant initiatives, e.g. creating language learning materials. the initial process of annotating speech for this purpose involves two tasks: segmentation and transcription, which we illustrate in figure 4 using two clips of jimmie's speech. in segmentation, the annotator identifies regions of speech and nonspeech and also which of the speech regions is english or muruwari. for a sequence of english sentence fragments such as those in clip a), the utterances can simply be merged into one. for mixedlanguage regions such as those in clip b), separate utterances should be created to allow the muruwari speech to be easily extracted for use in language learning materials. to create transcriptions for indexing, the annotator transcribes the english segments, given regions segmented and identified as english. we conducted a set of timed annotation tasks to evaluate to what extent the machineassisted workflow reduces the time taken to perform these two tasks. as detailed in the transcription task took 91.5 minutes of work for a 30-minute recording without machine assistance and on average 59.3 minutes when assisted (a 35% reduction). we found no meaningful difference between the correction times for transcriptions generated by asr systems with different levels of accuracy. for transcriptions produced by an off-the-shelf system (wer/cer: 36/22), the correction time was 63 minutes. for systems fine-tuned with 39 minutes of transcribed speech, wer/cer: 19/7 and 14/6, the correction times were 55.5 and 59.5 minutes, respectively. the closeness in transcription correction times may relate to how an english asr system whose wer is 30% or less produces good enough transcriptions for editing, according to a crowdsourced study @xcite . here, our transcribers' tolerance for the relatively less accurate off-the-shelf system (wer 36%) may be attributable to their familiarity with the speech domain and speaker @xcite , having collectively spent nearly 40 hours correcting transcriptions of jimmie's english by the time we conducted the timed tasks. these results suggest that, where correction is permissible by l1-speaking transcribers of the metalanguage, the time savings over manual transcription could still be gained using an off-the-shelf system that achieves a wer of 30-36% or less for the metalanguage in the recordings. nevertheless, we find that the machine-assisted workflow does offer time savings over a fully manual workflow (in line with previous work, e.g.: @xcite . specifically, we find that the machine-assisted workflow offers a 20% reduction in overall time to identify regions in the target language and metalanguage and also transcribe the latter, requiring 2.36 hours (82.5 + 59.3 mins) of correction time for a 30-minute recording compared to a fully-manual one which requires 2.95 hours (85.5 + 91.5 mins). unlike the manual workflow, the fully-automatable workflow can derive first-pass transcriptions to help an authorised person triage recordings. ## towards a muruwari orthography as mentioned above, the muruwari orthography is still currently in development. in this section, we provide a brief overview of how transcriptions of the english metalanguage are being used to aid in the development of the muruwari orthography. a key source of information on muruwari phonemes and words of interest to the current muruwari community are two 1969 recordings in which jimmie barker discusses an early muruwari wordlist @xcite . this wordlist was created by linguist r.h. mathews and consists of muruwari words in his romanisation along with english translations. using this wordlist, the documentation team is able to shortlist muruwari words whose romanisation is suggestive of containing sounds of interest (e.g. dental consonants), and then quickly locate in these recordings jimmie's pronunciation of the words and associated commentary using the time-aligned english transcripts generated for the two recordings. here, the english transcripts provide significantly more streamlined access to untranscribed muruwari utterances than browsing the recordings in real time. once verified of containing the sounds of interest, the documentation team is able to extract snippets of these words to be included in the community consultation process. ## conclusion many hours of unannotated speech from endangered languages remain in language archives and inaccessible to community members and language learning programs. the time-intensive nature of annotating speech creates one bottleneck, with an additional one occurring for speech in restricted access corpora that authorised community members must vet before annotation can begin. for a particular genre of recordings where speech in the endangered language is intermixed with a metalanguage in a more widely-used language such as english, we proposed a privacy-preserving workflow using automated speech processing systems to help alleviate these bottlenecks. the workflow leverages voice activity detection (vad) to identify regions of speech in a recording, and then spoken language identification (sli) to isolate speech regions in the metalanguage and transcribes them using automatic speech recognition (asr). the uncorrected transcriptions provide an estimate of the contents of a recording for an authorised person to make initial decisions on whether it can be listened to by those with lower levels of access to correct the transcriptions, which, collectively, help index the corpus. this workflow can be implemented using a limited amount of labelled data: 10 utterances per language for sli and 39 seconds of transcribed speech in the metalanguage for asr. the workflow reduces metalanguage transcription time by 20% over manual transcription and similar time savings may be achievable with an off-the-shelf asr system with a word error rate of 36% or less for the metalanguage in the target recordings. given our use case, the present demonstration of the workflow was limited to the scenario of processing single-speaker monologues with a mix of muruwari and english, the latter of which made possible the use of a state-of-the-art model trained for english asr (robust wav2vec 2.0: @xcite and also for transcriptions to be corrected by first language speakers of english. our work also revealed that vad and sli systems require further optimisation for mixed-language speech. we hope our demonstration encourages further experimentation with model adaptation with limited data for related use cases. for dialogues between a linguist and language consultant, for example, speaker diarisation could be added via fewshot classification using speech representations for speaker recognition (e.g. speechbrain sr embeddings: @xcite . with user-friendly interfaces like elpis @xcite , for which wav2vec 2.0 integration is underway (foley, pers. comm.), we hope to see more streamlined access to pre-trained models for language documentation workflows and, consequently, more streamlined access to the recorded speech for community members and language learning programs. morris swadesh. 1955. towards greater accuracy in lexicostatistic dating. international journal of american linguistics, 21(2):121-137. jörgen valk and tanel alumäe. 2021. voxlingua107: a dataset for spoken language recognition. in 2021 ieee spoken language technology workshop (slt), pages 652-658. ieee.