Local Word Discovery for Interactive Transcription (2021)

Abstract: Human expertise and the participation of speech communities are essential factors in the success of technologies for low-resource languages. Accordingly, we propose a new computational task which is tuned to the available knowledge and interests in an Indigenous community, and which supports the construction of high quality texts and lexicons. The task is illustrated for Kunwinjku, a morphologically-complex Australian language. We combine a finite state implementation of a published grammar with a partial lexicon, and apply this to a noisy phone representation of the signal. We locate known lexemes in the signal and use the morphological transducer to build these out into hypothetical, morphologically-complex words for human validation. We show that applying a single iteration of this method results in a relative transcription density gain of 17%. Further, we find that 75% of breath groups in the test set receive at least one correct partial or full-word suggestion.

## introduction in over a century of practice in descriptive linguistics, the pattern has been to prepare texts and a lexicon to support the construction of a grammar. the grammar includes a description of the phonology and morphosyntax, which inform the representation of the texts and lexicon, in a cyclic arrangement @xcite . the three types of data are entwined in the so-called "boasian trilogy" of texts, lexicon, and grammar. more recently, another tradition of working with little-studied languages has grown up in the language technology community. it frames these as "low resource languages," lacking the text, speech and lexical resources that are needed for creating speech and language technologies @xcite . in many cases, these languages are not little-studied at all, it is just that the technological methods can only exploit texts and lexicons, not the grammar. this brings us to the question: how can we leverage a grammar when working with a low resource language? in particular, how can we leverage a morphosyntactic description to accelerate the creation of texts and a lexicon for a morphologically complex language? our approach complements the practice of "learning to transcribe" @xcite , where nonspeaker transcribers train themselves to recognize words in connected speech. we assume that transcribers are able to sparsely annotate spans of audio with any words they recognize. these words can be aligned with the output of an automatic phone recognizer, and the machine suggests new words conditioned on phones in the locus of known words (fig. 1 ). we call this task local word discovery. in the case of low-resource languages like kunwinjku (iso gup), we do not have enough text to train a language model to guide the suggestion of words in the locus of previously recognized words. however, as a morphologically-complex language with a published grammar, we do have information at the level of morphemes. thus, we employ a morphological transducer to map previously recognized morphs with the surrounding noisy phone sequences to new morphologicallycomplex wordforms for manual verification. the constituent morphs of confirmed words are then added to the lexicon. figure 2 shows the proposed local word discovery pipeline, which we deploy in a prototype interactive transcription system. we test the system with speakers of kunwinjku. the main contributions are: a new word discovery task which cultivates a morph lexicon; a new, low-friction, interactive speech transcription workflow for low-resource morphologically-complex languages which leverages local word discovery; and a prototype implementation that integrates a universal phone recognizer with a morphological transducer. 1 . we begin with a review of related work (sec. 2), followed by an overview of the proposed task of local word discovery and our implementation of a model which performs this task (sec. 3). we then explain how we set up an evaluative experiment of the model (sec 4), and give results (sec. 5), followed by conclusions (sec. 6). ## previous work early work on computer-assisted speech transcription grew out of the increasing effectiveness of automatic speech recognition (asr) systems for resource-rich languages. for example, @xcite trained an asr system on 228 hours of transcribed speech from the national congress of japan. word recognition errors are manually corrected using various interfaces: multiple choice selection from confusion pairs, respeaking, and manual correction. subsequent work continues to build in human post-editing of increasingly accurate asr output @xcite . thanks to their reliance on asr, these systems depend on lexicons and large amounts of transcribed speech for training. the lack of performant asr systems for low-resource languages makes this approach ill-suited to the linguistic documentation use case; we can only automate the first stage of the asr pipeline, namely phone recognition. ## local word discovery this research takes place in the context of a series of engagements with the bininj community of west arnhem, in the far north of australia. the community is centered in the town of gunbalanya and a network of outstations, and predominantly speaks kunwinjku. schools, ranger programs and arts centres employ local people in cultural work where literacy in kunwinjku is considered desirable, though not yet well established. kunwinjku has limited electronic texts and lexicons, but there is a comprehensive grammar @xcite . transcription in this context is unavoidably collaborative, with a non-speaker transcriber working with a speaker and acquiring some of the language in the process @xcite @xcite . the non-speaker transcriber can transcribe familiar words in a first pass, and later prompt a speaker to produce any unrecognized words so they can be added to the lexicon and spotted automatically. over time, they become part of the vocabulary of the non-speaker transcriber, who is able to confirm their appearance more readily in future. such transcription work is held up by the presence of unknown words, disfluencies, coarticulation, and noise. it is wise to skip difficult passages at first, and transcribe words that can be easily recognized, only later coming back to fill in the gaps once the priorities for careful, contiguous transcription have been established. this practice has been called sparse transcription @xcite . sparse transcriptions become contiguous through iterative, interactive processes such as collaborative work with speakers, or by leveraging word spotting techniques to detect other instances of identified lexemes across a larger corpus. sparse transcription serves a number of realworld use cases aside from contiguous transcription, e.g. spotted words serve as an index into the audio, facilitating keyword-based retrieval across large corpora; and lexical entries and associated metadata can be used in language learning. ## experiment setup we explore the concept of local word discovery on sparsely-transcribed audio by measuring the change in transcription densities before and after applying local word discovery implemented with the fst. the first step is to define an initial lexicon which we use to sparsely transcribe a collection of audio. we used the collection of transcribed utterances from speaker sg as the test set, and the automatic phone recognition model which was fined tuned on all but sg's speech. from the transcriptions of sg's speech, we identify the 10 most frequent morphs and locate them in the speech. this is the input for word discovery (figure 3 , line i). this produced 126 annotated utterances: 126 breath groups represented by their phone stream, with individual tokens of the lexemes from the initial lexicon algorithm 1 finite state word discovery from sparsely transcribed input require: grammar the fst which transforms a valid surface string into its morph analysis 1: define wspace [..] (->) " "; optionally insert a single whitespace 2: define lexa [grammar .o. wspace].l; all surface forms optionally interspersed with a space 3: define lexb [0:" "] lexa [0:" "]; suppress space on either side of lexeme 4: define lexc [ "-":? ]* lexb [ "-":? ]*; recognize lexemes, padding all non-member characters 5: define lexd [lexc] .o. noisyphones; recognize lexc, flexibly allowing for phones in equivalence classes aligned to the speech. for each of the 126 lines of input, we calculated their baseline transcription density as the sum of character lengths of spotted lexemes divided by the sum of characters in the gold transcription. for example, if the utterance is "k a r i re", where "k", "a", "r", and "i" are phones and "re" is a lexeme, and its gold orthographic transcription is "karrire", then the baseline transcription density of this utterance is 2/7, or 28.6%. we ran each of the 126 sparsely transcribed utterances through the local word discovery pipeline defined in section 3.3. the output is a list of partial or full word completions, anchored in known lexemes (see figure 4 ). for each utterance, we examined the suggestions and accepted those that were correct based on the gold transcription (simulating the manual confirmation of a speaker). we report the transcription density increase relative to the baseline density, since the model seeks only to increase density around the locus of existing annotations. ## experiment results across the test set of 126 utterances, we found that 47.6% of them received correct, full word suggestions, and 75.4% received correct partial word suggestions. individual utterances varied widely in terms of baseline transcription densities, and how much local word discovery with an fst was able to contribute. in terms of characters transcribed solely by accepting full word suggestions anchored at the locus of known lexemes across all utterances in the corpus, we saw a transcription density growth of 17.34% relative to the baseline density. summary statistics on the performance of local word discovery on the sg collection can be seen in figure 8 . full word density is the number of characters transcribed before applying local word discovery plus the number of characters transcribed by accept- sg corpus baseline density (chars) 34.6% full word density (chars) 40.6% relative increase 17.3% % breath groups with full word suggestions 47.6% % breath groups with partial word suggestions 75.4% ing full word suggestions from the system output, over the the number of characters in the gold orthographic transcription. for individual utterances, we calculate the increase in transcription density, a subset of which can be seen in figure 9 . accepting only the full-word suggestions across the sg collection leads to the creation of 76 new unique entries for the lexicon. in the context of a full interactive transcription pipeline, this represents 76 new possible exemplars for lexeme spotting across the wider corpus. for reference, this experiment was seeded with just 10 unique glossary entries and produced more than 7 times that number of entries to seed a second round. as new instances of lexemes are confirmed, the local word discovery pipeline can be run again to discover more full-words around these new loci. ## conclusion the literature on low-resource languages has framed such languages as lacking the required texts and lexicons for developing the usual suite of speech and language technologies. recent work in this vein has generally not explored the use of published linguistic descriptions, the third linguistic data type in the boasian trilogy, perhaps because descriptions are seen to require too much manual labour to convert into computational grammars, or because the resulting grammars are seen to be too brittle for working with natural speech. nevertheless, we believe such descriptions can play a role in supporting the creation of texts and lexicons, while reducing the dependence on language models. a description, suitably interpreted, can constrain the forms that can hypothesized in a given textual context, and this information can be used to inform (rather than limit) the choices made by human transcribers. in this paper we have explored this idea and applied it to a morphologically complex language. we have proposed a new computational task of "local word discovery" to complement the practice of sparse transcription. we have discussed an approach to local word discovery that uses an existing morphological analyzer to process a sequence of known lexemes aligned to a noisy stream of phones. the method suggests possible completions of morphologically complex surface forms that are grounded at the locus of known lexemes and conditioned on the phonetic environment. on test data from kunwinjku, local word discovery increases transcription density by 17.3% and contributes 76 new unique glossary entries. these new entries then serve as new loci for further iterations of local word discovery. these results show that local word discovery a promising means of generating transcription suggestions which grow the lexicon and produce more dense transcriptions. this approach enables a novel transcription workflow where a non-speaker transcriber does a first pass, transcribing easily identifiable words, and a speaker comes along behind to work on the residue, while the system is performing word spotting and local word discovery in the background. we deployed the model in an interactive transcription system and tested it in the field and saw that local word discovery, together with the other stages of the new transcription workflow, enabled low friction interactions between transcribers and the system, speeding up the transcription process.