Explicit Tone Transcription Improves ASR Performance in Extremely Low-Resource Languages: A Case Study in B ribri (2021)

Abstract: Linguistic tone is transcribed for input into ASR systems in numerous ways. This paper shows a systematic test of several transcription styles, using as an example the Chibchan language Bribri, an extremely low-resource language from Costa Rica. The most successful models separate the tone from the vowel, so that the ASR algorithms learn tone patterns independently. These models showed improvements ranging from 4% to 25% in character error rate (CER), and between 3% and 23% in word error rate (WER). This is true for both traditional GMM/HMM and end-to-end CTC algorithms. This paper also presents the first attempt to train ASR models for Bribri. The best performing models had a CER of 33% and a WER of 50%. Despite the disadvantage of using hand-engineered representations, these models were trained on only 68 minutes of data, and therefore show the potential of ASR to generate further training materials and aid in the documentation and revitalization of the language.

## introduction the documentation and revitalization of indigenous languages relies on the transcription of speech recordings, which contain vital information about a community and its culture. however, the transcription of these recordings constitutes a major bottleneck in the process of making this information usable for researchers and practitioners. it typically takes up to 50 hours of an expert's time to transcribe each hour of audio in an indigenous language @xcite . moreover, there are usually few community members who have the expertise to transcribe this data and who have the time to do so. because of this, extending automated speech recognition (asr) to these languages and incorporating it into their documentation and revitalization workflows would alleviate the workload of linguists and community members and help accelerate their efforts. indigenous and other minority languages usually have few transcribed audio recordings, and so adapting data-hungry asr algorithms to assist in their documentation is an active area of research @xcite @xcite @xcite @xcite @xcite @xcite @xcite . this paper will examine an element that might appear obvious at first, but one where the literature is "inconclusive" @xcite , and which can have major consequences in performance: how should tones be transcribed when dealing with extremely low-resource languages? this will be examined by building asr models for the language bribri from costa rica. the results show that simple changes in the orthographic transcription, in the form of explicit tonal markings that are separate from the vowel information, can dramatically improve accuracy. ## transcription methodology the first step towards training an asr model in bribri was the selection of the training materials. the spontaneous speech corpus of flores solórzano (2017a) was used because of its public availability (it is available under a creative commons license) and because of its consistent transcription. this corpus contains 1571 utterances from 28 speakers (14 male and 14 female), for a total of 68 minutes of transcribed speech. these utterances contain a total of 13586 words, with 2221 unique words. the main question in this paper is: how can we easily reformat bribri text into the best possible input for asr? let's take the word dikì /diă£"ki ă £/ 'underneath' as an example. this word has two syllables, the first one with a low tone and the second one with a high tone, indicated by a grave accent. in addition to the tone, the second syllable is also nasal, and this is marked with a line underneath the vowel. one possible representation of this word would be to interpret it as four different characters, as is shown in condition 1 of ## traditional asr results kaldi was used to train models for each of the transcription conditions described above. two parameters were varied in the experiment: the number of phones in the acoustic model (monophone or triphone), and the number of words in a kenlm based language model (unigrams, bigrams and trigrams) @xcite . all other hyperparameters were identical to those in the default kaldi installation. thirty models were trained for each of the six transcription conditions, using the six parameter combinations (phones x ngrams), for a total of 1080 models. 3 to train these models utterances were randomly shuffled for every model and then split so that 90% of the utterances were used for training (1571 utterances) and 10% were used for validation (174 utterances). each of the models had two measures of error: the median character error rate (cer) and the median word error rate (wer), calculated over the input transcription for each condition. the results reported below correspond to the median of the 30 medians in each condition. figure 1 shows the summary of the training results. the condition with the best performance is tonesep, where the tone symbol is kept separate (high, fall, rise), the low tone is left out as a default, and the nasal feature remains connected to the vowel symbol (i.e.: a versus a-nas). tonesep is not the condition with the least vowel symbols, but it is the one with the best performance. this could be due to two reasons. first, what tone-sep appears to be doing is changing the behavior of the triphone window. kaldi's acoustic model has states with three symbols in them. in a writing system that only has graphemes for segments, the triphone window would, indeed, look at the consonant or vowel in question and to its preceding and following segments. with tonesep, the tone symbols are surrounded by the vowel the tone belongs to and the following consonant or vowel (or at the nasal symbol). this means that, in practice, when the triphone window looks at the tone, it is looking at two actual phones (the vowel, its tonal cues, and the following consonant/vowel), or even one actual phone (the vowel with its tonal and nasal cues). there are well known effects of tones in their preceding and following segments @xcite @xcite , so this reduced window might be helping the computer generalize the relatively stable tone patterns of bribri and their effect on the surrounding segments. the training chops the duration of the vowel into two segments; the first chunk is used to identify the vowel itself, and the second chunk is used to identify the tonal trajectory. 4 a second reason for the advantage of tonesep might be the phonetics of the low tone itself. it is not only the most frequent tone in bribri, but it also the least stable phonetically. the low tone can actually appear as low or mid, depending on its surrounding tones @xcite . what kaldi might be doing is simply learn the more stable patterns of the other tones and label all other pitch patterns as "low". the reason why tonenassepwl is the worst performing transcription is unclear. it might be the case that the addition of the low tone creates an explosion in the number of hmm states, given that the low tone is the most frequent one. another reason might be the separation of the nasal feature. it is possible that the nasal vowels of bribri are different enough from their oral equivalents that trying to decouple the vowels from their nasality makes generalization more difficult. as can be seen in figure 1 , the nassep condition also performs poorly. this pattern matches results in languages like portuguese @xcite and hindi @xcite , where the best results are obtained by keeping the nasal fea4 no experiment was conducted to test the effect of placing the tone indicator before the vowel (e.g. d low i k high i nas for dikì 'underneath'). in theory, the performance would be worse given that, in the early milliseconds of a vowel, tones can be phonetically co-articulated with their preceding tone and these two cues would blend together @xcite @xcite . this effect, called carryover, causes greater deformations in pitch than the effect of anticipating the following tone, or anticipatory assimilation @xcite . therefore, the second part of the vowel would provide a clearer tonal cue. ## end-to-end results end-to-end algorithms need massive amounts of data to train properly @xcite , so they are not the most appropriate way to train the small datasets characteristic of extremely low-resource languages. however, it would be useful to test whether the differences detected in the traditional asr training are also visible in end-to-end training. a ctc loss algorithm with bidirectional rnns was used, specifically that implemented in deepspeech. two types of endto-end learning were studied: first, models were trained using only the available bribri data. this style of training will be called just bribri. second, the bribri data was incorporated into transfer learning models @xcite @xcite . deepspeech has existing english language models, foot_2 trained with 6-layer rnns. the final two layers were removed and two new layers were grafted onto the rnn. the first four layers would, in theory, use their english model to encode the phonetic information, and the final two layers would receive that information and produce bribri text as output. removing two layers was found to be the optimal point of transfer learning, which matches previous results in literature utterance meaning: 'so you were young then, right?' target utterance: e' ta be' bák ia tsítsir wake' tonesep e'ta be' bák ia tsítsir wake' cer: 3% allfeats e'ta bák ia tsítsir wake' cer: 16% tonenassepwl e' ta wake' chìchi wa cer: 61% there might be several reasons why the situation has reversed in the ctc models. first, providing an explicit symbol for the low tone might force deepspeech to look for more words in the transcription. as can be seen in a third reason for this improvement might have to do with the size of the alphabet: the wl conditions have relatively few symbols for the vowels (12 symbols for tonenassepwl versus 48 for allfeats), which would result in a smaller output layer for the rnns. notice that, as with the triphones in kaldi, the rnns might be splitting the vowel into separate chunks. it would then proceed to identify the type of vowel from the first chunk, the tone in the second and the nasality in the final part. it would also benefit from the bidirectionality of the neural networks, finding tonal cues in the surrounding segments without the disadvantages of gmm/hmm systems. finally, it should be noted that the transfer models did not provide an improvement in performance. this is somewhat surprising; this might indicate that the bribri dataset is too small to benefit from the transfer, or that the knowledge of english phones does not overlap sufficiently with the bribri sound system to produce a boost. even then, the transfer models also show effects due to the different transcription conditions, and they also benefited from separating the tone and nasal features from the vowel. this effects will have to be confirmed in the future with other end-to-end techniques, such as listen, attend and spell algorithms @xcite and wav2vec pretraining @xcite . ## conclusions while hand-engineered representations are suboptimal for high-resource languages, these can still be helpful in low-resource environments, where they can help set up a virtuous cycle of creating imperfect but rapid transcriptions, which can then be improved to create more training materials, improve asr algorithms, and start helping documentation and revitalization projects right away. the results above show that performing relatively easy transformations in the input (e.g. not marking the most common tone, separating the tonal markings from the vowel) can lead to major improvements in performance. it also shows that nlp practitioners and linguists can fruitfully combine their knowledge to understand the different features involved in the writing system of a language. additionally, it provides evidence that the benefits of phonetic transcription can also be gained using semi-orthographic representations. the following recommendations provide a short summary of the results: (i) separate the tones from the vowels. this will help asr systems learn their regularities. (ii) experiment with other features, such as nasality; if they modify the formants of the vowel, they should probably be grouped with the vowel. finally, this work is the first attempt at training speech recognition for a chibchan language. as shown in