Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in T unisian Dialect (2024)

Abstract: Speech encoders pretrained through self-supervised learning (SSL) have demonstrated remarkable performance in various downstream tasks, including Spoken Language Understanding (SLU) and Automatic Speech Recognition (ASR). For instance, fine-tuning SSL models for such tasks has shown significant potential, leading to improvements in the SOTA performance across challenging datasets.In contrast to existing research, this paper contributes by comparing the effectiveness of SSL approaches in the context of (i) the low-resource Spoken Tunisian Arabic Dialect and (ii) its combination with a low-resource SLU and ASR scenario, where only a few semantic annotations are available for fine-tuning. We conducted experiments using many SSL speech encoders on the TARIC-SLU dataset. We used speech encoders that were pre-trained on either monolingual or multilingual speech data. Some of them have also been refined without in-domain nor Tunisian data through a multimodal supervised teacher-student learning. The study made in this paper yields numerous significant findings that we will discuss in the paper.

## introduction self-supervised learning methods aim to train a representational model, also called upstream model, that benefits a collection of downstream tasks. ssl models are trained by using information extracted from the input data itself as the label to target. various techniques have been introduced in the literature in order to learn powerful representations from the speech signal, including those based on autoregressive predictive coding @xcite , contrastive losses @xcite and masked prediction @xcite @xcite . other works explored the combination of contrastive learning and masked language modeling @xcite . the learned ssl models like wav2vec 2.0 @xcite , wavlm @xcite , data2vec @xcite , data2vec 2.0 @xcite , @xcite , w2v-bert @xcite have been proven effective for a wide range of tasks as they considerably lighten the amount of annotated speech data normally required for downstream tasks, while exploiting very large amounts of unlabeled data. some models have been extended to a cross-lingual setting through xls-r-128 @xcite , @xcite and the recently released w2v-bert 2.0 model @xcite . several efforts went further to enrich the frame-level speech representations with textual semantic information. in 2022, @xcite proposed such a model called samu-xlsr allowing the obtention of a better semantic encoding in the speech representations of a pretrained ssl model. samu-xlsr approach follows a teacher-student supervised learning framework that uses multilingual text/audio paired data and the language-agnostic bert sentence embedding (labse) model @xcite . this teacher-student approach is used to refine an ssl pre-trained model dedicated to speech processing to make it able to produce sentence-level embeddings from speech similar to the embbedings given by the labse model from the speech transcriptions. they use the commonvoice version 8 corpora applied to 53 languages to refine the xls-r-128 model @xcite . more recently, a multilingual sentence embedding space for 200 text and 37 speech languages called sonar @xcite was introduced by meta ai. the authors adopted a twostep training strategy: they first build a sentence embedding space for the multilingual text before extending this to the speech modality via a teacherstudent approach. applied to many downstream tasks, ssl speech encoders have shown their great potential by improving the state-of-the-art performances on challenging benchmark datasets. however, understanding speech encoder capabilities requires a benchmarking effort to compare and draw insights across the techniques. indeed, there has been a considerable amount of work and effort in order to benchmark such ssl models. superb @xcite is a good example of this effort. it provides a comprehensive speech ssl benchmark including tasks such as phoneme detection, asr, slot filling, intent detection, keyword spotting, etc. the superb benchmark has been extended to evaluate multilingual speech systems in the ml-superb benchmark @xcite . xtreme-s @xcite is another example that focuses only on ssl models trained with multiple languages. despite the large number of high-quality benchmarks that evaluate ssl models on various downstream tasks, a limited number of studies have probed their effectiveness for downstream tasks in spoken arabic dialects. in this paper, we propose to broaden the ssl benchmarking effort to the automatic speech recognition (asr) and spoken language understanding (slu) of spoken arabic dialects. the contributions of this work are as follows: ## benchmarking protocol in this section, we formulate the downstream tasks and describe the tunisian dialect dataset together with the speech encoders used in our study. ## discussion we carried out some error analyses to quantify some slu systems performance across different conditions. we focus on 6 ssl models: lebenchmark, xls-r-128, data2vec 2.0, samu-xlsr, w2v-bert 2.0 and sonar-arb. ## conclusion our study investigates the usage of various ssl speech encoders for a spoken language understanding task in challenging circumstances characterized by a scarcity of both slu and asr training data and the low-resource characteristics of the targeted tunisian arabic dialect. our findings emphasize the efficacy of ssl pre-trained speech encoders in such conditions, with notable success observed when employing the w2v-bert 2.0 model, with 600 millions parameters trained on 4.5m hours of speech from 143 languages. additionally, we highlight the noteworthy performance of data2vec 2.0, pre-trained on english monolingual data, particularly excelling in handling semantically complex utterances. these outcomes collectively provide valuable information for advancing slu methodologies, especially in resource-limited linguistic contexts. last, samu-xlsr provides very competitive results thanks to semantic enrichment made by the teacher student approach and in future work we plan to train a samu-w2vbert 2.0 model to take benefit of the joint samu and w2v-bert 2.0 capabilities.