To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation (2024)

Abstract: Arabic is known to present unique challengesfor Automatic Speech Recognition (ASR). Onone hand, its rich linguistic diversity andwide range of dialects complicate the de-velopment of robust, inclusive models. Onthe other, current multilingual ASR modelsare compute-intensive and lack proper com-prehensive evaluations. In light of thesechallenges, we distill knowledge from largeteacher models into smaller student variantsthat more efficient. We also introduce a novelhuman-annotated dataset covering five under-represented Arabic dialects for evaluation. Wefurther evaluate both our models and existingSoTA multilingual models on both standardavailable benchmarks and our new dialectaldata. Our best-distilled model’s overall perfor-mance (45.0% WER) surpasses that of a SoTAmodel twice its size (SeamlessM4T-large-v2,WER=47.0%) and its teacher model (Whisper-large-v2, WER=55.1%), and its average perfor-mance on our new dialectal data (56.9% WER)outperforms all other models. To gain more in-sight into the poor performance of these modelson dialectal data, we conduct an error analysisand report the main types of errors the differentmodels tend to make. The GitHub repositoryfor the project is available at https://github.com/UBC-NLP/distill-whisper-ar.

## introduction there have been significant advancements in multilingual automatic speech recognition (asr) in both training methodologies and architectures. models such as openai's whisper @xcite and meta's seamlessm4t @xcite can transcribe speech from languages in the order of the hundreds, albeit with varying degrees of accuracy. especially for low-resource languages, these models do not perform well @xcite @xcite ). arabic, for example, poses significant challenges to these multilingual models and hence is the object of the current work. arabic can be classified into three broad categories, namely: classical arabic (ca), used in early literature and religious texts; modern standard arabic (msa), the 'high' variety used in official documents and in the media; and dialectal arabic (da), the collection of 'low' varieties used in day-to-day conversations @xcite . da can vary extensively at the regional level (e.g. gulf vs maghrebi), country level (e.g. egyptian vs sudanese), and sub-country (e.g. hourani or northern jordanian dialect vs urban or madani dialect) @xcite @xcite . due to the significant differences in lexicon, phonetics, and even grammar between these varieties, asr systems trained on msa alone cannot be reliably leveraged off-the-shelf for all arabic speech. developing effective models for da can prove especially difficult, given the lack of standardized orthography, the scarceness of labeled data for many dialects, inconsistent use of diacritics, and use of code-switching @xcite . although most multilingual and multimodal systems (e.g., @xcite @xcite ) cover arabic, their evaluation predominantly involves benchmarks established for msa, such as fleurs @xcite , common voice (cv) @xcite , and the arabic speech corpus (asc) @xcite . since arabic exhibits substantial linguistic diversity, encompassing various varieties and dialects, evaluations conducted solely on msa are inherently limited. existing works aiming to address this gap, e.g., @xcite , lack thorough evaluation and do not cover current state-of-the-art (sota) models. to address this, we conduct a comprehensive evaluation of all recently developed models on a linguisti-cally diverse set of arabic datasets. beyond the challenge of inadequate evaluation, the deployment of massive multilingual multimodal systems such as seamlessm4t @xcite and whisper @xcite is hampered by the considerable computational resources they require during both training and inference. these efficiency issues pose a significant accessibility barrier, discriminating against populations with limited resources. to alleviate this concern, we employ a framework for knowledge distillation @xcite from large models such as whisper @xcite into relatively compact models for arabic speech recognition. we show that our distilled models are not only compute-efficient but their performance is on par or better compared to larger counterparts. in summary, the gaps in existing work include @xcite the insufficient knowledge about the utility of recent multilingual speech model models on arabic, including dialects, (2) the discrepancy in representing some arabic dialects in existing dialectal benchmarks, and (3) the inefficiency of these models due to their large sizes which demands significant compute resources at both training and inference time. we address these limitations through a number of contributions, as follows: the rest of the paper is organized as follows: section 2 is a review of related works. in section 3, we introduce knowledge distillation and outline our related methods and training strategies. in section 4, we provide details about our experiments, and in section 5 introduce and discuss our results. section 6 delivers a thorough error analysis based on the model predictions on our new dialectal data. we conclude the work in section 7. finally, we oultine our limitations and ethical considerations in sections 8 and 9, respectively. arabic asr. for arabic, the performance of these models remains under-explored. while ope-nai's whisper model whisper-large-v3 @xcite achieves 15.1% word error rate (wer) on common voice 15.0's @xcite arabic split and 9.6% wer on fleurs @xcite , a performance close to humanlevel, @xcite show that it is vulnerable to linguistic variations where its performance degrades substantially on several arabic dialects. furthermore, the performance of other multilingual systems such as seamlessm4t @xcite , universal speech model (usm) @xcite , and xls-r @xcite on diverse arabic varieties remains unknown. efficiency. the size of these massive multilingual systems poses another challenge to their usability. to address this, @xcite shows that speculative decoding @xcite can expedite the generation from whisper by a factor of two. efficient transformer inference engines such as ctranslate2 (opennmt) based inference sys-tran can also improve the generation speed, despite having the same memory requirements. moreover, although quantization techniques have been effective in reducing memory requirements @xcite , they do not decrease the number of active parameters, leading to variable improvement in generation speed @xcite . knowledge distillation. knowledge distillation is a method used to transfer knowledge from large models to smaller ones, thereby reducing both memory and compute requirements @xcite @xcite @xcite . this technique has been effectively applied in various domains. for example, in computer vision applications, knowledge distillation results in compact and efficient models @xcite . similarly, in diffusion models @xcite and large language models @xcite , knowledge distillation produces small, efficient, and task-specific models. yang et al. (2023) distill knowledge from multiple foundation models into small and dedicated speech recognition models. @xcite proposes cross-modality knowledge distillation from large language models into speech models. knowledge distillation in speech. ferraz et al. (2024) distill knowledge from a large whisper model into small multilingual models but limit their evaluation to standard benchmarks in eight languages (arabic not included in the set). @xcite apply a novel distillation approach to whisper, reducing its size by 80-90% while also improving its performance. @xcite propose a layer-wise distillation approach that reduces the size of a hubert model by 75% while increasing its processing speed by 73%, retaining most of the original model's performance across multiple tasks. in addition to that, researchers have introduced methods for model compression, such as data-free knowledge distillation and teacher-student (ts) learning for domain adaptation @xcite . these approaches involve training student models to mimic teacher models using various strategies, including gaussian noise generation and sequence-level kullback-leibler (kl) divergence @xcite . among different knowledge distillation approaches such as the ones highlighted above, the standard student-teacher distillation is a task-and modality-independent framework that is simple yet effective. @xcite use this framework to distill whisper into small monolingual models for english using large-scale pseudo-labels. however, their work is limited to high-resource language. we take inspiration from @xcite and distill whisper into small models for arabic and perform a thorough evaluation. one dif-ference between our work and that of @xcite is that while there is limited information about the out-of-distribution datasets of @xcite 's work and whether they are part of the teacher's training data, we employ new dialectal speech data never seen by the model. ## knowledge distillation knowledge distillation is a method of transferring knowledge from a large model (teacher) to a relatively small model (student). the student model is trained to mimic the behavior of the teacher model both at the dense representation level and the sequence level @xcite @xcite . following @xcite , who distill a whisper model for english asr, we first generate large-scale pseudo-labels from the teacher model and apply a threshold to filter the output. we then train the student model with high-quality filtered pseudo-labels as ground truth, which can be expressed as: the student model is also trained to minimize the discrepancy between the probability distributions over tokens of the student and teacher models, based on kl divergence: objective: we take the weighted sum of ( 1 ) and we use the same values for α kl (0.8) and α p l (1.0) as @xcite . details about our teacher and student models, along with training data, can be found in in other work for training and evaluating arabic asr models @xcite . upon inspection of the data, we found it to be composed of mostly msa along with some ca speech. multi-genre broadcast. multi-genre broadcast (mgb) @xcite is a challenge for a wide range of arabic speech understanding tasks such as speech recognition, speaker identification, dialect identification, etc. we experiment with three variants, namely mgb2, mgb3, and mgb5. mgb2 has roughly around 70% msa, with the remainder containing other dialects @xcite . ## results and discussion we evaluate all models on four versions of cv (6.1, 9.0, 11.0, 15.0), mgb-2, mgb-3, mgb-5, fleurs, and our five novel dialectal sets. we report wer and cer scores on the orthographic and normalized predictions (as per section 4.3) in commercial systems and supervised models. the supervised finetuned (sft) baselines are trained on mgb-2, mgb-3, and the cv datasets. other evaluation sets thus represent outof-distribution data. as a result, we see that supervised hubert (15.4) and whisper (25.2) outperform all other models on in-distribution data mgb-2 and mgb-3, respectively. however, these baselines often perform poorly on all other evaluation sets that are not in their training data. on our private in-house data, the supervised models usually produce more incorrect words than the number of words in the corresponding reference. we find varying levels of transcription difficulty for these models when evaluated on distinct dialects and linguistic varieties. the amazon transcribe system performs well on our in-house data compared to the supervised baselines. it gives 45.5% wer on jor which is not too far from the best wer of 41.5% by seamlessm4t-large-v2. we find that it struggles with alg, which goes along the trend noticed with all models. zero-shot models. we find that both whisper and seamlessm4t models perform quite well on cv 3 and fleurs in zero-shot setting. more specifically, the best whisper model shows wer scores of 15.8% and 11.3% on cv and fleurs, respectively, while the best seamlessm4t achieves 9.7% and 7.6% wer. mms shows the lowest performance of the zero-shot models and the second lowest across all model types with an 82.5% average wer. meanwhile, a consistent challenge across all models is observed with mgb-5, followed by mgb-3. these last two datasets involve dialects; namely, egy and mor dialects respectively. the transition from msa to dialects thus marks a signif3 we refer to cv15.0 as cv. as expected, we observe that increasing the number of layers in the distilled model enhances its performance. consequently, our best-performing distilled model, dw-32-16++ (wer 45.0%), surpasses all other models, including whisper-large-v3 (wer 49.5%) and seamlessm4t-v2 (wer 47.0%), despite being half of its size (see to sum up, our best-performing distilled models yield the best results in terms of wer on four out of ten evaluated datasets and are on par with an overall best model in terms of average wer while being half in size. however, when looking at the average performance across in-house data only, it outperforms all other systems with a 56.9% wer, whereas the best zero-shot model (seamlessm4tlarge-v2) has 61.74% wer and teacher model average is the mean score across all the evaluation sets. all distilled models are trained with a filtering threshold of 80. we report the score on the test split of each dataset. abbreviations. w -whisper, ft -finetuned, m -medium, l -large, s -small, d -distil. our results underscore the distilled models' inherent efficiency and generalization to unseen dialects, possibly resulting from the mixture of data. this may imply that the process retains the critical linguistic and acoustic features necessary for high-quality asr in a linguistically diverse setting. ## error analysis to gain a better understanding of the results, we conduct an error analysis on our in-house data by randomly sampling 20 sentences per dialect from each models' outputs, with the aim of identifying the specific types of errors present. we then categorize the errors into the following types: msa translation: the transcription is semantically accurate but employs words in msa that differ from the dialectal words spoken in the utterance. hallucination: the transcription is found to be both semantically and acoustically distant from the utterance. deterioration: the transcription is either gibberish (random characters) or involves an excessive repetition of the same word or expression. incomplete transcription: parts of the utterance are omitted and do not appear in the transcription. ## conclusion we present a comprehensive evaluation of multilingual asr systems on a wide range of arabic varieties and dialects to assess the robustness and generalization capability of these systems to linguistic variations. we then distill small dedicated models for arabic asr from large multilingual speech models (whisper). we evaluate our distilled models on ten diverse datasets and find that despite being 25-50% smaller, they outperform the base model and are on par with state-of-the-art models twice their size. we also find our distilled models to be the most robust to linguistic diversity. we further conduct a comprehensive error analysis to investigate the nature of the errors these models make. we find that speech models with language model decoding are more prone to hallucination compared to other models. our work reveals an inherent limitation of these models to generalize beyond their training data. in the future, we intend to expand this work to low-resource and unseen languages.