Enabling Interactive Transcription in an Indigenous Community (2020)

Abstract: We propose a novel transcription workflow which combines spoken term detection and human-in-the-loop, together with a pilot experiment. This work is grounded in an almost zero-resource scenario where only a few terms have so far been identified, involving two endangered languages. We show that in the early stages of transcription, when the available data is insufficient to train a robust ASR system, it is possible to take advantage of the transcription of a small number of isolated words in order to bootstrap the transcription of a speech collection.

## introduction in remote aboriginal communities in australia, many efforts are made to document traditional knowledge including rock art, medicinal plants, and food practices. while it may be relatively straightforward to capture spoken content, transcription is time-consuming and has been described as a bottleneck @xcite . transcribing is often seen as an obligatory step, to facilitate access to audio. efforts have been made to speed up this process using speech recognition systems, but the amount of data available in indigenous language contexts is usually too limited for such methods to be effective. recent research has shown the efficacy of spoken term detection methods when data are scarce @xcite . taking advantage of the transcription of a few words would allow us to propagate it through the speech collection and thus assist language workers in their regular transcription work. so-called "sparse transcription" would be also a way to navigate a speech collection and allow us to be selective about what needs to be transcribed @xcite . several tools exist for manual transcription, such as elan and praat @xcite . however such transcriptions are often made in isolation from the speech community (first languages australia, 2014), and so we miss out on the opportunity to take advantage of the interests and skills of local people to shape and carry out the transcription work. we present a fieldwork pipeline which combines automatic speech processing and human expertise to support speech transcription in almost-zero resource settings. after giving the background, we detail the workflow and propose a pilot experiment on two very low-resource corpora. ## background existing approaches to automatic transcription of endangered languages involve methods that have been developed for automatic speech recognition. while a few hours of transcribed speech can be enough to train single-speaker models @xcite , speaker-independent models require a large amount of training data to produce useful transcriptions @xcite . moreover, they draw language workers and speakers into the time-consuming task of exhaustive transcription, forcing them to transcribe densely, including passages that may be difficult or impossible given the early state of our knowledge of the language. a more suitable approach, we believe, involves beginning with stretches of speech where we have the greatest confidence, and only later tackling the more difficult parts. spoken term detection involves retrieving a segment in a speech collection, given an example. with this method, it is possible to sparsely transcribe the corpus, i.e., take advantage of an existing transcription (or a list of spoken words) and identify tokens throughout the collection @xcite . dynamic time warping (dtw) @xcite and its more advanced versions @xcite can be used to match a query in a speech collection, but they can be computationally expensive. dtw "aligns two sequences of feature vectors by warping their time axes to achieve an optimal match" @xcite . a common method is to compute dtw, sliding the spoken query over each utterance in the corpus. in addition, several speech representations have been considered in the literature for the task of word-spotting. @xcite and @xcite explore feature extraction from autoencoders (ae) and correspondence autoencoders (cae) and show how top-down constraints can improve the quality of the hidden representation for spoken term detection tasks. @xcite introduce wav2vec, a self-supervised model for speech representation learning which is based on contrastive predictive coding, and apply this to supervised asr. ## results: impact of speech feature representations in the workflow we report the results of this new workflow in speaker diversity in the two corpora (sec. 4) has an impact on the final results. for mboshi 70% of the words to be retrieved in the speech collection are pronounced by the same speaker in the final lexicon and 37% for kunwinjku. figure 2 reports the proportion of tokens retrieved between same speakers and different speakers over the total number of tokens retrieved for mboshi and kunwinjku. the first observation is that the pipeline mostly retrieves terms pronounced by the speaker. it is clear that mvn improves performance for both same-speaker and different-speaker retrieval. we also note that, while wav2vec representation is overall less performant, it seems promising for extracting more speaker-independent representations, as illustrated by the results obtained with w2v en mb features. future work will investigate this by leveraging more raw speech in mboshi and kunwinjku for training better self-supervised (wav2vec) models. regarding false positives ( query false positive query translation hit translation nahne mahne it (pronoun) this (demonstrative) nemekke namekke that one that one (other spelling) balanda balanda-ken white man of the white man (genitive) bininj bininj-beh man from the man nemekke yekke that one dry season mahni mahne this (demonstrative) this (other spelling) ## deployment building on prior work developing mobile tools for language documentation @xcite , we have begun to explore methods for deploying the pipeline in a remote community. while the first step of identification of new words is straightforward, the task of lexical confirmation might be much more complex to apply. the members of an aboriginal community might not be familiar with technologies and they are not necessarily literate (in the narrow western sense). taking into account these constraints, we built a lexical confirmation app and trialled it on a small lexicon (fig. 3 ). the idea would be to load the output of the spoken term detection system into the app. then a speaker can listen to the query, listen to the utterance where we expect the query to be found, and confirm if the utterance contains the query, (fig. 1c ) ## conclusion we investigated the use of spoken term detection methods as an alternative to the usual methods that have been inspired by automatic speech recognition, and which require exhaustive transcription even for passages which exceed the present state of our knowledge about the language. instead we devised a workflow based on spoken term detection, and simulated it for two small corpora: one in mboshi, one in kunwinjku. the simulations of this workflow show that, with well chosen speech representations, we may have a viable approach for rapidly bootstrapping transcriptions of large collections of speech in endangered languages. the next step of this work would be to design methods to involve indigenous people in tasks such as the construction of the lexicon or the confirmation of the output of our system for a deployment of this workflow in a remote community.