Efficient Speech Translation through Model Compression and Knowledge Distillation (2025)

Abstract: Efficient deployment of large audio-language models for speech translation remains challenging due to their significant computational requirements. In this paper, we address this challenge through our system submissions to the ‘Model Compression’ track at the International Conference on Spoken Language Translation (IWSLT 2025). We experiment with a combination of approaches including iterative layer pruning based on layer importance evaluation, low-rank adaptation with 4-bit quantization (QLoRA), and knowledge distillation. In our experiments, we use Qwen2-Audio-7B-Instruct for speech translation into German and Chinese. Our pruned (student) models achieve up to a 50% reduction in both model parameters and storage footprint, while retaining 97-100% of the translation quality of the in-domain (teacher) models.

## introduction multimodal foundation models have shown powerful capabilities in different tasks, including speech translation. however, these models are often large and computationally intensive, making them impractical to use in real-world settings with limited resources. to enhance the efficiency of these models, researchers have been investigating diverse approaches to model compression that aim to reduce the computational requirements while retaining performance @xcite @xcite . qwen2-audio @xcite is a stateof-the-art foundation model that accepts various audio signal inputs and performs audio analysis or direct textual responses to speech instructions. in the iwslt 2025's model compression track @xcite , the organizers required that all submissions must be derived from the qwen2-audio model. the official languages of the task are english-to-german (en-de) and english-to-chinese (en-zh). we experimented with various approaches including efficient fine-tuning using quantized lowrank adapters with 4-bit quantization (qlora) @xcite , iterative layer pruning based on layer importance evaluation @xcite @xcite , and sequence-level knowledge distillation @xcite @xcite . our experiments are mainly based on qwen2-audio-7b-instruct. 1 while section 3 elaborates on our experiments, we can summarize our two submissions for german and chinese as follows: ## methodologies we experiment with diverse methods to compress qwen2-audio-7b-instruct while maintaining the translation quality. this section covers our two main experimental setups, while section 4 dis-cusses several ablation studies and elaborates on intermediate experiments. our first experimental setup (cf. section 3.2) employs qlora fine-tuning with 4-bit quantization, while the other experimental setup (cf. section 3.3) conducts layer importance evaluation and applies iterative layer pruning of the model, followed by qlora fine-tuning. both setups use knowledge distillation to recover the translation quality of the in-domain teacher model. ## ↓ ## ablation study this section elaborates on some intermediate experiments that led us to the final models in section 3.2 and section 3.3. these experiments include comparing the performance of the qwen2-audio base model with qwen2-audio-instruct (cf. section 4.1), comparing encoder-decoder pruning with decoder-only pruning (cf. section 4.2), comparing "iterative" layer pruning with fixed middle-layer pruning (cf. section 4.3), iterative pruning of up to 16 layers (cf. section 4.5), fine-tuning before and after pruning (cf. section 4.6), and using different sizes of out-of-domain data to fine-tune the pruned models (cf. section 4.7). ## inference and evaluation for inference, we use greedy generation by disabling the sampling options. we apply the prompt illustrated in section 4.1. we use a batch size of 1, and set the generation max length to 1024 tokens. to evaluate our systems, we calculated bleu @xcite , chrf++ @xcite , as implemented in the sacrebleu library foot_6 @xcite . while we use chrf++ for german, we use raw chrf for chinese, following the author of the metric who noted that "the concept of chinese words is generally not clear" @xcite . for semantic evaluation, we use comet (rei et al., 2020). 8 ## conclusions and future work in this work, we showed that combining multiple compression techniques enables substantial model size reduction with minimal impact on speech translation performance. to conclude, qlora finetuning with knowledge distillation achieved superior translation quality compared to layer pruning alone, though with reduced model compression. to achieve higher compression ratios while preserving translation quality, we employed a combined approach using iterative layer pruning, quantization, knowledge distillation, and multi-stage fine-tuning. the code of our experiments is publicly available. 9 in future work, we plan to explore adaptive compression strategies that dynamically adjust pruning levels and quantization precision based on realtime deployment constraints such as memory limits and latency requirements. additionally, we aim to evaluate our compression techniques across more diverse datasets, including both authentic and synthetic training data, to better understand the generalization capabilities of our approach. given that qwen2-audio-instruct relies on text prompts for generation, it would be interesting to investigate retrieval-augmented generation with few-shot prompting to improve the translation quality of compressed models.