Phone Based Keyword Spotting for Transcribing Very Low Resource Languages (2021)

Abstract: We investigate the efficiency of two very different spoken term detection approaches for transcription when the available data is insufficient to train a robust speech recognition system. This work is grounded in a very low-resource language documentation scenario where only a few minutes of recording have been transcribed for a given language so far. Experiments on two oral languages show that a pretrained universal phone recognizer, fine-tuned with only a few minutes of target language speech, can be used for spoken term detection through searches in phone confusion networks with a lexicon expressed as a finite state automaton. Experimental results show that a phone recognition based approach provides better overall performances than Dynamic Time Warping when working with clean data, and highlight the benefits of each methods for two types of speech corpus.

## introduction efforts are made across australia to preserve, document and revitalize aboriginal languages. these languages exist primarily in spoken form, and even if there often is an official orthography available, it is not widely used by local people. making recordings of speakers has been a widespread practice for documenting traditional knowledge. however, such recordings are often not transcribed, making them hard to access. manual transcription is time consuming and is often described as a bottleneck @xcite . while automatic speech recognition (asr) has seen great improvements in recent years @xcite , it relies on a large amount of annotated data. attempts to build asr systems for low-resource languages end up with high word error rate or single-speaker models making them of limited use in indigenous contexts @xcite . such methods assume that everything should be transcribed. @xcite describes a sparse transcription model where we only transcribe the words we can confidently recognize, using word-spotting, while leaving the transcription of more difficult sections for later, perhaps when a speaker is available @xcite . based on this model, @xcite proposed a workflow which combines spoken term detection and a human-in-the-loop to support transcription in under-resourced settings. such a workflow avoids the use of a language model which requires too much textual data, data that we cannot find in most aboriginal contexts, and which only needs a few spoken terms to be annotated. while they show through their simulation the capability of iterative transcription in remote communities, the precision of their method depends on the quality of the spoken queries, and the density of the resulting transcription is limited by the size of the lexicon. automatic phone recognition has seen progress with minimal data @xcite . while bird (2020) argues that phonetic transcriptions do not stand in for the speech data and cannot be segmented to generate the required higher-level word units, we can nevertheless view phone transcriptions as a speech encoding, retaining our commitment to the sparse transcription model. such an approach has an advantage over traditional query-by-example methods in that a simple word list can be used instead of a spoken lexicon which can be challenging to collect. in this paper we show how this can be done, and compare it with dynamic time warping (dtw) @xcite commonly used for keyword spotting for indigenous languages. we consider both methods as applied to two very low-resource languages, kunwinjku (gup) spoken in the far north of australia and mboshi (mdw) spoken in congo brazzaville. traditional asr systems are not well suited to aboriginal languages. the lack of data for such languages does not allow us to train an acoustic model or a language model. additionally, the type of data usually recorded is often spontaneous and noisy which makes it difficult to transcribe, regardless of the amount of annotated data available. @xcite describes the sparse transcription model, which combines spoken term detection with a human-in-the-loop, in an iterative process. using spoken term detection as a transcription method allows us to avoid traditional components of an asr system, specifically the language model, to focus only on the recognition of isolated words. traditional spoken term detection systems rely on text-based search in lattices extracted from asr systems @xcite . attempts to train asr systems in lowresource contexts have so far provided poor results for single speaker systems @xcite . this makes traditional spoken term detection approaches questionable in very low-resource settings. a few papers linked to the babel project have explored lattice search using asr systems trained in low-resource settings @xcite . however, they work with much larger data collections than what is available in indigenous contexts. query-by-example methods have been preferred in very low-resource contexts since they only rely on acoustic comparison between spoken queries and utterances. le ferrand et al. (2020) explore feature representation using dtw in an iterative pipeline following the sparse transcription model @xcite , and have been able to transcribe up to 42% of a lexicon in their speech collections. this method, however, has shown limitations in terms of robustness in the face of speaker variability. research around speech features for spoken term detection has explored the use of bottleneck features, or the hidden representation of an auto-encoder @xcite @xcite . such research highlights the benefits of multilingual approaches for spoken term detection when transcribed data are limited in the target language. others have exploited neural approaches to train word classifiers from word pairs using a siamese loss @xcite , however pairs of words are required, limiting the selection to words that can be searched. query-by-example relies on a spoken lexicon and, by extension, a comparison between two acoustic vectors. a difference of speakers or recording channel between the query term and the speech collection has an influence on the likelihood of a given term to be retrieved. moreover, a spoken lexicon is not simple to gather and this therefore limits the amount of terms we can retrieve. using a lexicon made of terms recorded in isolation for spoken term detection purposes will lead to poor precision. another solution would be to manually extract the terms of the lexicon from a speech collection which is time-consuming. phone recognizers, like asr systems, also need a few hours of annotated speech to provide acceptable performance @xcite . however, recent work has shown how multilingual phone recognizers can be fine-tuned with minimal data to work on a new language @xcite . raw phone transcriptions are hard to obtain as they require the skills of a trained linguist, and they cannot help directly for retrieving higher level-units @xcite . however, the orthography of most indigenous languages is based on their phonology and there is usually a simple mapping from graphemes to phonemes can be obtained to train a phone recognizer, even with a shallow knowledge of the phonology. a spoken term detection method based on a phone recognizer could allow us to rely only on written queries following a traditional lattice-search method. ## methods we begin with a lexicon of size s consisting of audio clips of spoken words, along with orthographic transcriptions, plus a speech collection in which more instances of those words may be found. two spoken term detection approaches, involving a multilingual component, are investigated here: (a) a baseline method based on dtw applied on multilingual bottleneck features (mbnf); and (b) a method based on a textual search in phone confusion networks extracted from a universal phone recognizer (p2w). ## data we are using a corpus of spontaneous speech in kunwinjku built from several sources. the training, validation and test set are described in we are also using a corpus of mboshi speech which consists of 4.5 hours of speech elicited from text with orthographic transcription and a forced alignment at the word level @xcite . training, validation and test sets have been extracted from the corpus and are described in the lexicon queries (for spoken term detection) are made of 100 words for mboshi and 60 words for kunwinjku. we randomly selected in the test set words which occur at least 3 times in the corresponding corpus. for each word, we manually selected examples clearly pronounced, respecting the speaker distribution of the test set ( ## combining the methods we mentioned in section 2 that dtw and p2w each have their own strengths. as we know, dtw will cope more easily with spontaneous speech and co-articulation effects such as assimilation and elision. phone recognition allows us to avoid gathering spoken queries and retrieving terms with exact matching between written forms. to highlight the complementarity of the methods, we analyse the intersection of their true positives in figure 8 . we show that across both corpora the intersection of the true positives is small, and so combining the two methods can help us increase the coverage of the transcription to reach up to 49.99% for kunwinjku and 32.16% for mboshi. we analysed the most common terms retrieved by dtw which have been ignored by p2w. for kunwinjku, the glottal stop and doubled consonant are the phones the least properly recognized (wanjh written wanj kunwardde written kunwarde for example). more generally, since the data used in kunwinjku is spontaneous speech, most of the missed hits by p2w are due to highly mistaken phone transcriptions by allosaurus. for mboshi, beyond the main easily-confusable phones (o / ω, e / for instance) the main missed hits are due to tones or long vowels not being correctly recognized. the baseline provides a match for every utterance/query pair if no threshold is applied. however, since p2w is restricted by the phones output by the phone recognizer, we have a limited amount of candidates regardless of the threshold. as mentioned before, this has the advantage of being more precise, but can easily miss a match if the phone lattices contain many mistakes. in view of this, we combine the two methods as follows. for each utterance/query pair brought by p2w, we first keep for evaluation the candidates which have a score greater than the p2w threshold. then we send to evaluation every pair having a distance less than the dtw threshold. we provide in the described way of combining the methods outperforms both p2w and dtw approaches in terms of f-score. for mboshi, we can observe a small increase of the recall with a precision barely affected. for kunwinjku, the results are less clear. while the f-score outperforms both the baseline and p2w, combining the methods double the recall but decreases by half the precision. ## conclusion this paper compares two methods of spoken term detection, one based on dtw with bottleneck features, and one based on on phone recognition. both methods have been applied on two very lowresource languages, namely, a corpus in mboshi recorded in a controlled environment, and a corpus of spontaneous speech in kunwinjku recorded in remote communities. experimental results shown that a few minutes of transcribed speech can be used to fine-tune a universal phone recognizer. then searching terms in a confusion network with a lexicon expressed as a fsa outperforms the baseline for mboshi but not for kunwinjku. a text-based approach has the advantage over traditional query-by-example that a set of written queries is easier to gather than spoken queries. further analysis has shown that the proposed phone recognition approach is more robust to speaker variability and tends to be more accurate than dtw overall. however, the baseline seems to have a better coverage over the corpora and to be more suitable with noisy data. one method relies on canonical orthography while the other relies on acoustic comparison. both methods have their own benefits depending on the type of data they are applied to. experimental results have shown that it is possible to take advantage of both methods to increase the overall recall while maintaining precision at an acceptable rate.