Automatic Speech Recognition and Query By Example for Creole Languages Documentation (2022)

Abstract: We investigate the exploitation of self-supervised models for two Creole languages with few resources: Gwadloupéyen and Morisien. Automatic language processing tools are almost non-existent for these two languages. We propose to use about one hour of annotated data to design an automatic speech recognition system for each language. We evaluate how much data is needed to obtain a query-by-example system that is usable by linguists. Moreover, our experiments show that multilingual self-supervised models are not necessarily the most efficient for Creole languages.

## introduction there is a long tradition of description of creole languages since, at least, the pioneering work of hugo schuchardt . creole languages have sometimes been assigned a special role in linguistics: as a type of 'mixed languages', they are often considered as illustrating a break in language transmission and do not fit the generally assumed historical/genetic tree model foot_0 . meanwhile they remain, for many of them, under-resourced languages. gwadloupéyen, spoken mainly on guadeloupe island (france) by around 700.000 speakers, is a vigorous but largely under-equipped and underresourced language. morisien, (mauritius island) is spoken by approximately a million speakers. these two languages still suffer from a low social status and remain mainly spoken languages (rather than written). the cream project aims at providing linguists with new methods for computational language documentation: automatic speech recognition and keyword-spotting (query-by-example, @xcite . the cream project teams up field lin-guists with computer scientists in order to address this low resource challenge. our contribution is twofold: 1) we introduce new methods for creole language documentation based on a combination of automatic speech recognition and keyword-spotting (in particular query-byexample qbe) ; 2) our experiments shed new light on some key assumptions about creole languages, i.e. its distance from the lexifier language foot_1 . gwadloupéyen (gcf) and morisien (mfe) are two frenchbased creole languages. this category groups languages that share french as the most important part of their lexicon, but have a significantly different grammar. we illustrate this phenomenon in (1-a.) and (1-b.) for gcf and mfe respectively, where most of the words (if not all) are clearly identifiable by a french speaker despite the difference in orthography: while these languages are well studied and vigorous, they are mostly spoken languages used in context of a dominant language: french for gwadloupéyen (see @xcite , french and english for morisien (see @xcite , a.o.). since they are spoken in two distinct linguistic and geographic areas (lesser antillean island of guadeloupe for gwadloupéyen and mauritius, in the indian ocean, for morisien), there is no contact between these two languages, which makes them an interesting case study for a comparison (no con-tact 3 , lexicon based on french, different grammars). ## transcribing creole languages since gwadloupéyen and morisien are mostly spoken languages, their written form is not stable. there are resources such as dictionaries and grammars for both languages @xcite @xcite @xcite , but writing in creole and transcribing spoken speech are two separate tasks. in the context of diglossia, code-switching is very frequent (see @xcite @xcite and obviously causes problems for an automatic transcription task. we focused here mainly on gwadloupéyen and we identified three main problems with the transcriptions available in @xcite . first, several words are transcribed in two different forms: anko vs ankò 'again', apré vs après 'after', bitin vs biten 'thing'. second, the transcriber hesitates between a transcription in french or in creole: and last, the transcriber chose to transcribe in the proper creole form (identified as basilectal) while the speaker pronounced a word quite similar to its form in french: transcribed dantis but pronounced as in french dentiste. creole languages are known to have a large range of variation, often described as the 'creole continuum', (see @xcite @xcite . this fact has even been theorized as a historical evolution towards the lexifier language, but see mufwene (1997); aceto 3 and no mutual understanding @xcite . in order to efficiently correct these errors and to allow the linguist to search for a word (i.e. a segment of speech) in the corpus independently of its transcription, we designed an experiment of keyword spotting (qbe). this task is in line with @xcite , and is brought into action when there is a need for the linguist to verify or correct the transcription. speech processing for creole languages has not received much attention so far. for gwadloupéyen, @xcite is, to our knowledge, the only relevant work in nlp, but it does not address speech recognition. for haitian creole, breiter (2013) explores speech recognition but haitian and gwadloupéyen are clearly distinct languages. for morisien @xcite is a recent initiative for creating a creole speech engine. however, it does not seem to address the same tasks as this work. ## asr with self-supervised learning self-supervised learning (ssl) is the task of learning powerful representations from huge unlabeled data (called pretraining) to recognize and understand patterns from a less common problem (called fine-tuning). recent work focused on speech data have reported impressive results for representation learning, and more specifically improved performance on downstream tasks for asr in lowresource contexts @xcite . these work are based on the wav2vec2.0 @xcite model. in our approach, we consider 2 models : 1) xlsr-53 @xcite , a multilingual pretraining of wav2vec2.0 model on 53 languages with more than 56k hours of unlabeled speech data (xlsr-53) which has been shown to construct better speech representations for cross-lingual transfer @xcite ; 2) lebenchmark @xcite , a french-based wav2vec2.0 model with the assumption that these creoles are closely related to french. ## query by example query by example (qbe) consists in detecting specific words in speech recordings thanks to the use of speech recognition approaches. keywords are defined according to the user's request. within the scope of this work, our keyword spotting approach firstly uses self-supervised learning models to predict the word in a speech segment. in the second phase, it searches for the prediction in a set of transcriptions. pre-processing for fine-tuning each audio recording is segmented into small segments, each corresponding to a sentence. audio segments are mono, with a sampling frequency of 16 khz. the pre-processing of textual data involves the deletion of punctuation marks, and a harmonization of specific characters (lowercase, the substitution of accentuated vowels such as 'à' into 'a', 'ê' into 'e', ...). for each experiment, we split the data into train, validation and test sets with a ratio of 80/8/12. details about the datasets are given in appendice a.1.1. ## discussion field linguists from the cream project evaluate very positively the results given in moreover ## conclusion and perspectives of course, an asr system cannot solve the problems that the human transcribers have not solved, i.e. the choice of transcribing a word in french or in creole (code-switching or not) 6 . our results show that qbe can complement asr and provide an easy way to scan the corpus for relevant examples. we found that a model pretrained on french performed better for gwadloupéyen and morisien than a model pretrained on a large typological set of languages 7 . for future work, we intend to apply the same method on english-based creole languages (such as jamaican creole) and portuguese-base creoles (kriol of guinea-bissau), to allow for a comparison and a generalization. pascal vaillant and isabelle léglise. 2014. à la croisée des langues. annotation et fouille de corpus plurilingues. revue des nouvelles technologies de l'information, pages 81-100. donald winford. 1997. re-examining caribbean english creole continua. world englishes, 16(2):233-279. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, remi louf, morgan funtowicz, joe davison, sam shleifer, patrick von platen, clara ma, yacine jernite, julien plu, canwen xu, teven le scao, sylvain gugger, mariama drame, quentin lhoest, and alexander rush. 2020. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, online. association for computational linguistics. a appendices a.1 appendice a: implementation details a.1.1 datasets corpus train dev test oov (%) gcf 68 4 8 24.68 mfe 52 3 5 24.82