u D istil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes (2025)

Abstract: Recent work on distilling Whisper’s knowledge into small models using pseudo-labels shows promising performance while reducing the size by up to 50%. This results in small, efficient, and dedicated models. However, a critical step of distillation using pseudo-labels involves filtering high-quality predictions and using only those during training. This step requires ground truth labels to compare with and filter low-quality examples, making the process dependent on human labels. Additionally, the distillation process requires a large amount of data thereby limiting its applicability in low-resource settings. To address this, we propose a distillation framework that does not require any labeled data. Through experimentation, we show that our best-distilled models outperform the teacher model by 5-7 WER points and are on par with or outperform similar supervised data filtering setups. When scaling the data, our models significantly outperform all zero-shot and supervised models. Our models are also 25-50% more compute- and memory-efficient while maintaining performance equal to or better than that of the teacher model. For more details about our models, dataset, and other resources, please visit our GitHub page: https://github.com/UBC-NLP/uDistilWhisper .

## introduction to democratize automatic speech recognition (asr), significant attention has been given to multilingual models @xcite @xcite . these powerful systems, thanks to their massive number of parameters and training data, can simultaneously transcribe in hundreds of lan-guages. that being said, low-resource languages tend to trail behind in performance compared to high-resource counterparts such as english @xcite . for instance, while openai's latest asr model whisper-large-v3 shows the word error rate (wer) in the single digits on english and spanish, a much lower performance is shown for east asian and african languages 1 @xcite . while the performance reported on these multilingual models is not as low for arabic, it is important to highlight that it does not fully and accurately represent the diversity of the different varieties that characterize the arabic language family @xcite . in addition to the disparity in performance, multilingual models are extremely compute-intensive and are thus not equally accessible to everyone. in order to alleviate this issue, knowledge distillation has been the method of choice for multiple works @xcite @xcite . knowledge distillation has proven to be highly effective in reducing model size, thereby lowering compute and memory requirements while maintaining performance comparable to larger teacher models. for example, @xcite and @xcite demonstrate its effectiveness for english and arabic, respectively. however, a significant limitation in these works is their reliance on ground-truth labels to filter out low-quality pseudolabels generated by the teacher model-a resource that is often scarce, particularly for low-resource languages. additionally, @xcite shows that training on unfiltered data leads to suboptimal performance. this reliance on labeled data underscores the need for an unsupervised approach to data filtering in knowledge distillation, enabling distilled models to perform better without depend-ing on any ground-truth labels. in this work, we propose frameworks to distill knowledge from teacher models that do not rely on filtered pseudo-labels. our method achieves a similar performance to supervised distillation and outperforms pseudo-label distillation without filtering. our contributions are the following: we organize the paper as follows: we present a review of the existing literature in section 2 before introducing knowledge distillation along with our methodology in section 3. section 4 contains particulars of our experiments and section 5 delves into the results we obtain. we discuss these results in section 5, conclude in section 6, and outline our work's limitations in section 7. ## related work scaling speech foundation models in both data and model size has led to systems capable of handling a wide range of speech tasks @xcite @xcite . these models effectively transcend language barriers, demonstrating robust performance across numerous languages, including many low-resource ones @xcite . however, these models face two key limitations: (1) their large size @xcite , which results in high computational costs, and (2) the under-representation of low-resource languages in training, leading to suboptimal performance on these languages @xcite . to address the first issue various efficient decoding methods @xcite @xcite were proposed. however, resulting models do not necessarily enjoy a reduced memory footprint. to address the efficiency issue, knowledge distillation @xcite is proven to be very effective. different approaches for knowledge distillation have been explored such as patient knowledge distillation (pkd) @xcite , t-s learning @xcite , cross-modal hashing (cmh) @xcite , joint unsupervised domain adaptation with kd @xcite and data-free kd @xcite among others in the nlp and cv fields alike. knowledge distillation has also been applied to speech recognition: @xcite shrink a whisper @xcite model tangibly (∼ 80% of its original size), all while improving its performance; @xcite reduce a hubert model to 75% of its initial size through layer-wise distillation without a significant drop in performance. in addition, knowledge distillation has found successful applications in speech-to-text tasks as well, @xcite @xcite effectively reducing the memory and compute requirements. more recently, it has been used to distill multilingual models from whisper @xcite and strong monolingual models @xcite , as well as in lowresource settings. an example of the latter is waheed et al. (2024) who investigate pseudo-label distillation methods @xcite across various modern standard arabic (msa) and dialectal arabic datasets. their exploration demonstrate the efficacy of distillation in enhancing both efficiency and performance, showcasing that smaller dedicated models can outperform larger multilingual ones. while the pseudo-labeling approach has proven effective across diverse languages @xcite , it traditionally requires labeled data for filtering low-quality pseudolabeled examples. in this work we address this limitation by introducing an unsupervised framework based on data filtering methods, thereby eliminating the dependency on labeled data altogether. ## experiments for a fair comparison and analysis, we keep our experimental setup quite identical to @xcite . apart from evaluating our models on five standard benchmark datasets, we extend the evaluation to include the sada @xcite and casablanca foot_4 @xcite datasets. we describe all the used datasets below. ## results and discussion we report the baseline results from waheed et al. ( 2024 ) and evaluate all models on sada's test and validation splits. our main results can be found in while larger variants of both seamlessm4t and whisper perform well on standard benchmark setups like common voice and fleurs, we find that they perform poorly on dialectal speech. for example, the best zero-shot model (seamlessm4tlarge-v2) has a 7.6% wer on fleurs compared to 65.0% on the sada test set as shown in when it comes to our distilled models, we observe that the two best filtering measures, sonarsim and proxy-ref, are comparable to or surpass the equivalent supervised setups. for instance, dw-32-16 trained on 100k examples with a filtering threshold of 80 achieves a wer of 35.3% on the benchmark test split, 66.6% on the sada test split, and 61.1% averaged on five novel dialects. by comparison, our best-performing model in a similar configuration, udw-32-16-sonar, achieves 35.5% wer on the benchmark test split, 58.7% on the sada test, and 60.6% on five dialects. this reflects close to 3% improvement in average score compared to the supervised setup (51.6 vs 54.3) and more than a 7% improvement over the no-filter setup, as detailed in ## conclusion in this study, we explore methods for distilling large whisper models into smaller, more efficient ones without relying on labeled data. our filtering techniques bridge a gap in prior research and facilitate the creation of compact and effective speech recognition models for limited label settings. we show through a comprehensive evaluation that our models outperform both their teacher model and those using supervised distillation. our evaluation spans a diverse range of arabic varieties, demonstrating their generalization to linguistic diversity and their competitive performance with sota models twice their size. applying our approach to swahili datasets further validates its effectiveness for different languages. notably, our model-based filtering methods (proxy and sonar) demonstrate superior robustness across linguistic variations. moving forward, we aim to explore model-free approaches to further enhance the efficacy of model distillation, while including extremely low-resource languages and domains.