Fine-tuning pre-trained models for Automatic Speech Recognition, experiments on a fieldwork corpus of Japhug (Trans-Himalayan family) (2022)

Abstract: This is a report on results obtained in the development of speech recognition tools intended to support linguistic documentation efforts. The test case is an extensive fieldwork corpus of Japhug, an endangered language of the Trans-Himalayan (Sino-Tibetan) family. The goal is to reduce the transcription workload of field linguists. The method used is a deep learning approach based on the language-specific tuning of a generic pre-trained representation model, XLS-R, using a Transformer architecture. We note difficulties in implementation, in terms of learning stability. But this approach brings significant improvements nonetheless. The quality of phonemic transcription is improved over earlier experiments; and most significantly, the new approach allows for reaching the stage of automatic word recognition. Subjective evaluation of the tool by the author of the training data confirms the usefulness of this approach.

## introduction the use of transformer-type neural architectures to learn multilingual models of text and speech, coupled with methods for fine-tuning these generic representations, has opened up the possibility of developing tools for the many languages for which there is only a small amount of annotated data available. this approach has special appeal for linguistic documentation tasks: the development of semi-automatic or even automatic transcription and annotation methods based on a small amount of annotated data would reduce the annotation effort of field linguists and language workers, who could then focus their attention on linguistically and relationally meaningful tasks during fieldwork @xcite @xcite . in this multidisciplinary endeavour, it is clear that "linguists and natural language processing (nlp) scientists may want to adjust their expectations and workflows so that both can achieve optimal results with endangered data" @xcite . the present work reports on our experiments using a pre-trained model of speech, xls-r @xcite , to develop a phonemic recognition system for a minority language of china: japhug (ethnologue language code: jya, glottolog code: japh1234; see @xcite . the transcription of recordings in a newly documented language is a key task for fieldworkers (linguists and language workers). it is also an interesting topic for the speech processing community, as it raises several challenges, epistemological as well as practical. first of all, the amount of data available for such languages is very small: for instance, of the 197 languages in the pangloss collection @xcite , which hosts audio recordings in various languages of the world (most of them endangered), only 44 corpora contain more than one hour of recordings. there is therefore a need for speech recognition methods that require as little training data as possible. in this respect, japhug can be considered as an outlier, since there is a 32-hour transcribed corpus, freely available in the pangloss collection 1 as well as from zenodo @xcite 2 and as a huggingface dataset. 3 the size of this corpus is one of the reasons for choosing japhug as the test case for the present investigations: we wanted to be able to evaluate the amount of data that is necessary to obtain an automatic transcription of good quality -an important criterion here being the linguist's evaluation of the usefulness of the automatically generated transcript, as will be discussed again below. research in the field of resource-constrained automatic speech recognition (asr) has brought out "the importance of considering language-specific and corpus-specific factors and experimenting with multiple approaches when developing asr systems for languages with limited training resources" @xcite . to mention two such factors: the presence of various loanwords, as well as cases of code-switching in the recordings, are a challenge for the automatic transcription of linguistic fieldwork data. conversely, there is one aspect in which automatic transcription tends to be easier for fieldwork data than for widely studied languages: namely, 1 https://pangloss.cnrs.fr/corpus/ japhug 2 https://doi.org/10.5281/zenodo . 5521111 3 https://huggingface.co/datasets/ benjamingalliot/pangloss their high degree of orthographic transparency. most endangered languages are languages transmitted through oral tradition, without a widely used writing system, and the transcriptions are usually made by linguists and language workers either in the international phonetic alphabet or in an orthography that is very close to the pronunciation. thanks to this last characteristic one may realistically hope to achieve good quality transcriptions, as the system does not have to learn a complex spelling -unlike in the case of orthographies which have less straightforward correspondences between graphemes and phonemes (e.g. uralic languages in cyrillic orthography have a high degree of grapho-phonematic complexity, raising some technical difficulties @xcite . the sections below are organized as follows: we start out, in section 2, by briefly describing the model we have used. then we move on to presenting, in section 3, the results of a first set of experiments on phonemic transcription, which show that xls-r does indeed allow us to produce very good quality transcriptions from a small corpus of annotated data, and that these transcriptions meet a need from the linguists conducting language documentation and conservation work. however, a second set of experiments described in section 4 shows that this result is difficult to reproduce, which leads us to qualify our initial optimistic conclusion concerning the technological dimension of the work. 4 2 fine-tuning pre-trained models principle the approach implemented in this work is based on the fine-tuning of a multilingual signal representation model, a method introduced in the field of speech recognition by @xcite to build speech recognition models from little data. this approach is today at the core of many nlp models and is considered by many to be the most promising way to develop nlp and speech systems beyond the thirty or so languages (representing only 0.5 % of the world's linguistic diversity) for which there are large amounts of annotated data @xcite . the proposed approach is composed of two steps. in the first step, xls-r, 5 a multilingual model trained in an unsupervised way on a corpus of 56,000 hours of recordings in 53 languages, is used to automatically build a language-independent, 'generic' representation of the signal. in a second step, this representation is used as input to a phonemic recognition system, trained on audio data that are time-aligned with a manual transcription provided by the linguist. this second step allows to learn how to match the signal representations with labels: in this case, it is essentially the labels corresponding to the phonemes. in our experiments, we used the xls-r multilingual model foot_2 and the huggingface api @xcite to use and fine-tune it. we ran the fine-tuning for 60 epochs (i.e. 60 iterations over the training data) to be assured that the fine-tuning had converged, and we kept the last model. using the model for phoneme prediction in order to apply the method described in the previous paragraph to the task of phoneme recognition, we simply defined a set of labels corresponding to the set of characters composing the phonemes. more precisely, the set of labels used for fine-tuning is made of the 44 characters that appear in at least one japhug phoneme. 7 this technical choice is based on the experiments reported by @xcite showing that the prediction of the characters composing the phonemes (instead of the phonemes as units) allows to obtain good predictions, sidestepping the task of explicitly listing the phonemes of the language (for example to specify that /ʈʂʰ/ constitutes a single phoneme, noted by a trigraph: ʈ+ʂ+ʰ). for the sake of simplicity at an initial exploratory stage, we also removed from the manual transcriptions all the punctuation marks and the other miscellaneous symbols used by linguists in their transcriptions (symbols to note linguistic phenomena of emphasis or focus, for example). to this set of grapho-phonemic labels is added the space, to delimit words, thereby coming a step closer to the development of a true speech recognition system for endangered languages. the addition of a special character marking the word boundaries is a novelty in our work; foot_4 it aims at allowlarge-100k-voxpopuli. ## evaluation on the japhug language in order to facilitate the reproduction of the experiments, the japhug corpus is made available as a huggingface dataset 9 which can be used off-theshelf with the tools described here. ## conclusion in this work, we have described how the fine-tuning of a multilingual model could be used to learn an automatic phonemic transcription system for an endangered language, and thus reduce the annotation effort of field linguists. despite the large variability of the scores obtained on a validation set, we succeeded in developing systems whose predictions required only a small number of manual corrections by the linguist: a number that is much smaller than that estimated by the character error rate (cer). this work shows the interest of this type of approach, and opens many perspectives. in particular, the approach seems to us to call for an extension of the experiments to other endangered languages (e.g. from other corpora hosted in archives of endangered languages, about which see berez-kroeker and henke 2018), in order to evaluate more widely its usefulness for language documentation. we also wish, in our future work, to improve the quality of predictions at the word level, for example by integrating a language model.