A Finite State Aproach to Interactive Transcription (2022)

Abstract: We describe a novel approach to transcribing morphologically complex, local, oral languages. The approach connects with local motivations for participating in language work which center on language learning, accessing the content of audio collections, and applying this knowledge in language revitalization and maintenance. We develop a constraint-based approach to interactive word completion, expressed using Optimality Theoretic constraints, implemented in a finite state transducer, and applied to an Indigenous language. We show that this approach suggests correct full word predictions on 57.9% of the test utterances, and correct partial word predictions on 67.5% of the test utterances. In total, 87% of the test utterances receive full or partial word suggestions which serve to guide the interactive transcription process.

## introduction thousands of the world's languages have small populations and are characterized by primary oral usage @xcite . these local languages co-exist alongside trade languages, i.e., languages of commerce, education, mass media, and government. local languages are generally losing ground to larger languages, a process known as language shift @xcite . key features of local languages are that they generally have no literary tradition, and little incentive exists for writing. there is often no established or widely known orthography, and usually no widely accepted standard variety to render into writing. the point where a related dialect becomes a distinct language may not be clearly understood or widely agreed. many heritage communities seek to reclaim or revitalize their ancestral languages @xcite . here, people often depend on historical sources, including informal collections of audio recordings, in order to access the ancestral code. scholars are also involved, using historical recordings in the process of language documentation and description @xcite . ideally, everything would be transcribed, and it would be easy to access the content of such collections for the purposes of learning and scholarship. however, given that these are oral languages, there is usually no pool of readily available transcribers to call upon. none of the above is systematically addressed by current low-resource approaches to transcription, which require upwards of 100k words (or 12-27 hours) of training data in the language, in order for sufficiently accurate phone recognition to support reasonable word error rates. such work generally assumes that a comprehensive lexicon is available, and we find that this is generally not the case. we seek a new approach, one that works with the locally available resources and human capacities. our work is founded on three insights. first, work on indigenous languages proceeds from locally meaningful, locally motivated activities. this usually prioritizes content over form, interpreting over transcribing @xcite . two important use cases are language learning and accessing the content of media collections. we devise tasks that leverage informal linguistic knowledge, such as the ability to form morphotactically valid words, and specialized knowledge of the vocabulary that pertains to a semantic domain of interest. this insight does not simply connect with local motivation, it is an effective way to meet the reciprocity requirement for ethical indigenous research @xcite . second, work with speakers of indigenous languages is more effective when it involves collaboration on realistic tasks. thus, we operate within the skill set and time availability of speakers and linguists. in particular, we eschew artificial tasks like phonetic transcription, instead tapping into people's ability to identify words in connected speech @xcite . this can in-volve learning vocabulary, and getting clear about nuances of word meaning by drawing on usage in context, or speech concordances. third, we apply what is known about the language, even when it is a non machine readable grammar, by interpreting it into a computational form that can be deployed to guide language tasks. thus, our contribution is a novel approach to transcribing local languages that is: locally motivated, feasible, and leverages what is already known about the language. as proof of concept we provide a finite state implementation, using the framework of constraint-based optimality theory @xcite . we envisage that this implementation, suitably optimized, could be deployed in an interactive, collaborative, sparse transcription system. ## available resources low-resource languages are not necessarily understudied; many have significant description. this work benefits from an existing finite state morphological analyzer @xcite . we use it as an acceptor of morphotactically valid strings in the language, combining canonical lexemes with noisy phone recognizer output. additionally, it is common for linguists to maintain a bilingual lexicon, and a corpus of up to 10k words of human-transcribed speech. the computational model described in the following section incorporates these resources. we define two lexicon classes: "topical" words, semantically relevant to the audio we are transcribing, and "attested" words, those known to exist in the overall corpus. finally, recent advances in phone recognition have made it possible to train or fine-tune models capable of producing phone sequences from audio @xcite . allosaurus is a pretrained universal phone recognizer which allows for language-specific fine-tuning. we obtained the fine-tuned model of @xcite and used it to automatically generate noisy phone sequences from field recordings of kunwinjku speakers. ## joint alignment and local word discovery the goal of the proposed local word discovery model is to give useful signal to the transcriber in the form of full word suggestions-which may be completely or partially correct-conditioned on known lexemes provided by the transcriber. equally, we would like the model to be able to provide high confidence suggestions when possible, and back off to cast a wider net when necessary. in this section we propose a finite state implementation of local word discovery which accomplishes this, while also incorporating implicit alignment. the gen function takes a phone string and an ordered list of known lexemes, converts them to fsts, and produces a list of candidate strings marked for constraint violations. the eval function converts these candidate strings to fsts, and passes them through a cascade of constraints, implemented as fsts and combined using lenient composition (see fig. 3 ). we employ three types of constraint: (a) anchored -these constraints are anchored to the beginning or end of the phone string; (b) topical -a lexical constraint consisting of words already discovered in the recording we are transcribing; (c) attested -a lexical constraint consisting of words which are attested in the language. 1 we give more detail about the function of each of these components in the following sections. the python implementation is available foot_1 . ## model evaluation and results the objective of this model is to provide a reasonable set of word candidates which lead to correct transcriptions. as such, any model suggestions which correctly predict subword units beyond the anchor lexemes can be useful for helping the transcriber discern the full word they are hearing, as they interactively poll the model. accordingly, we chose to evaluate the performance of this model by automatically simulating a first pass at transcribing 126 utterances of the test set. these 126 utterances are recorded audio segmented by breath group, from a tour of kabulwarnamyo, conducted in kunwinjku. the input of the model requires a phone string, and an ordered list of known lexemes. as already mentioned, we use the allosaurus model fine-tuned for kunwinjku of @xcite . similarly, we adopt their sparse transcription data preparation method: we simulate a sparse transcription of the audio by selecting a vocabulary of the top 20 morphs occurring in the training set, and use that vocabulary to manually annotate the test set. to see how this works, suppose that the test set includes an audio file (2). the corresponding prepared sparse transcription would be the unaligned, ordered list of morphs from the "known" vocabulary, i.e., birri, balanda, birri. using these sparse transcriptions and the automatically derived phone strings, we fed the test set to the local word discovery model to generate a list of candidate words anchored at the locus of the known lexemes. in this way, we found that 12.7% of all predictions across all utterances were correct full word predictions. additionally, 38.2% of all model suggestions were partially correct, i.e., a substring of the suggested word attached to the anchor segment was correct, and thus a useful signal for the transcriber to decide how to continue transcribing the word (fig. 7 ). on the utterance level, 57.9% of result sets contained correct full word suggestions, and 66.7% contained correct partial word suggestions. in total, 86.5% of utterance-level result sets contained correct full or partial word suggestions. a sample of these results can be seen in fig. 6 . ## discussion the key feature of this model is that we drop manual lexeme-to-phone alignment, and instead perform alignment on the fly, incorporating newly identified lexemes. this is illustrated in (1), where newly identified morphemes for each iteration are marked in red. this innovation has an important consequence for the transcription process: it can be iterative. each time the linguist and/or speaker revisit an utterance, they consider a new set of suggestions for building the transcription out from known lexemes where the model has taken care of working out where everything fits. they may also posit entirely new lexemes and add them to the lexicon. for each new lexeme, the user only needs to indicate their relative position with respect to existing lexemes. for each new visit to an utterance, the lexicon is in an expanded state due to transcription of other utterances, and the model makes new suggestions. the model handles some subtle issues in transcription. for example, when a morph appears multiple times in an utterance, as we see for kabirri "they" (1). when a transcriber adds a lexeme, the model assigns it to the best location, but only for the purpose of discovering words anchored at this lexeme (1a). when the transcriber identifies a new lexeme, e.g. durrkmirri "work", the previously number of utterances 126 % predictions full correct 12.7 % predictions partial correct 38.2 % docs with full correct 57.9 % docs with partial correct 66.7 % docs with any correct 86.5 identified lexeme is aligned elsewhere (1b). further suggestions-by the human transcriber or an automatic word spotter-identify a second instance of kabirri (1c). alongside these benefits are some shortcomings. first, if a transcriber is mistaken about the identity of a lexeme, the model will not be able to come up with better suggestions for that locus, except in the unlikely event that there is another locus where that incorrect lexeme can be aligned. second, the model may generate suggestions for a given anchor lexeme, when a user wants to work on a different part of the utterance. here, the user may need to accept high priority suggestions (if they are correct) and wait for a later iteration to get model suggestions for other parts of the utterance. third, thanks to the iterative nature of this approach, the precision and recall of the model for a given utterance depends on how high we are in the constraint hierarchy when results are returned. high priority constraints are more precise, with results sets of 1 or 2 candidates (varying on the size of the topical lexicon). low priority constraints contain edit distance-based variations on the source signal, and therefore can grow quite large with as a function of uncertainty. this variability with precision and recall leads to a further benefit. the model is able to prioritize precision when possible, while backing off to recall when necessary. accordingly, for our test data, the average number of predictions per utterance is just 6.5, compared to an average of 64.1 predictions per utterance of the non-constraint based model of @xcite . a further shortcoming of our approach is that we must compile unique fsts at runtime. this means we cannot precompile the network with lexc and foma or hfst compilers, but must use python bindings, and compile fsts dynamically with each new input. this could be prohibitively slow in some instances, as complexity increases exponentially with the size of the phone stream and known segment lists. a solution is to add a preprocessing step: utterances are already segmented from the original audio using silence; any overlong utterances are further split on confirmed full words. ## conclusion we have proposed a novel approach to collaborative transcription, which works with locally available resources and human capacities. in particular, local indigenous participation is not reduced to laborious and unmotivated phone transcription, but focuses on the identification of keywords in connected speech. these may be relevant to a concurrent cultural activity, or to language learning, in which the meaning of words in context is of more interest than their phonemic representation. the results suggest that this model does well in leveraging a computational grammar to give meaningful, interactive signal in a collaborative transcription context. this model improves on previous local word discovery models in that it is able to suggest words while performing alignment implicitly. we anticipate that this approach will integrate with interactive, collaborative transcription systems, such as @xcite . we also hope to have shown a way of bridging language data collection to locally-motivated language work.