BUINUS at IWSLT : Evaluating the Impact of Data Augmentation and QL o RA -based Fine-Tuning for M altese to E nglish Speech Translation (2025)

Abstract: This paper investigates approaches for the IWSLT low-resource track, Track 1 (speech-to-text translation) for the Maltese language, focusing on data augmentation and large pre-trained models. Our system combines Whisper for transcription and NLLB for translation, with experiments concentrated mainly on the translation stage. We observe that data augmentation leads to only marginal improvements, primarily for the smaller 600M model, with gains up to 0.0026 COMET points. These gains do not extend to larger models like the 3.3B NLLB, and the overall impact appears somewhat inconsistent. In contrast, fine-tuning larger models using QLoRA outperforms full fine-tuning of smaller models. Moreover, multi-stage fine-tuning consistently improves task-specific performance across all model sizes.

## introduction despite increasing advances in multilingual technologies, the development of speech translation (st) systems for low-resource languages continues to pose significant challenges. maltese, though an official language of the european union, exemplifies these difficulties. currently, there are approximately 200 language resources available for maltese, a relatively small amount, especially compared to the availability of resources for languages spoken in more populous countries @xcite . with fewer than one million speakers and a scarcity of both transcribed speech and parallel text corpora, maltese remains underresourced in the context of speech and language processing. this paper describes our approach to the iwslt 2025 low-resource shared task for the maltese-english language pair. speech translation involves two main components: transcription and translation. for transcription, we primarily fine-tune whisper @xcite , while for translation, we fine-tune nllb @xcite . for the larger nllb model, we also incorporate qlora @xcite , one of the best parameter-efficient fine-tuning methods, to accommodate resource constraints @xcite . however, we treat transcription mainly as a supporting infrastructure and focus the majority of our experimentation on the translation component. data augmentation techniques have become indispensable in machine translation, particularly for addressing the challenges posed by limited parallel data in low-resource languages @xcite . the term "low-resource" refers to the limited availability of data for one of the languages-in this case, maltese. a common strategy to mitigate this issue is data augmentation @xcite , which aims to reduce the likelihood of the model encountering completely out-of-distribution data during translation @xcite . unlike most approaches that augment data by generating similar text, the method proposed in @xcite introduces auxiliary tasks such as token swapping, sentence reversal, and the insertion of unk tokens to enhance model performance. we found this approach promising and adapted it slightly. specifically, we fine-tuned the nllb model in two stages: first on both auxiliary tasks and the main translation task, and then on the main task alone to finalize the model. ## system overview our speech translation pipeline comprises three main components: transcription, machine translation, and data augmentation. each component is optimized to address the specific challenges of low-resource translation settings. ## results model size and fine-tuning strategy. our results indicate that the larger 3.3b nllb model, fine-tuned using qlora, outperforms the smaller 600m model that is fully fine-tuned. while the larger models achieve higher overall performance after fine-tuning, this may partly reflect its stronger baseline performance. the performance gain from fine-tuning is actually greater for the smaller 600m model, suggesting that smaller models benefit more directly from the fine-tuning process, while larger models rely more on their pretrained capacity. effect of data augmentation. for the 3.3b model, none of the tested data augmentation techniques such as paraphrasing, token swapping, unk token insertion, or sentence reversal led to noticeable gains, with the highest improvement being just 0.001 comet points from paraphrasing. the 600m model, on the other hand, showed slightly better results, with consistent but small improvements across all methods, reaching up to 0.0026 comet points. while the gains for the smaller model are more apparent, they remain modest. these results suggest that data augmentation may be more useful for smaller models, which benefit more from the added variability due to their limited capacity. this aligns with prior findings @xcite , where augmentation strategies had a greater effect on less-pretrained models. ## limitations the limitation of our study is the lack of extensive qualitative analysis due to limited language proficiency. since we do not fully understand the language in the dataset, our analysis primarily relies on quantitative methods. ## conclusion in this paper, we explore the use of pretrained models-whisper for asr and nllb for mt-alongside data augmentation and parameterefficient fine-tuning methods. our experiments show that fine-tuning larger nllb models using qlora outperforms full fine-tuning on smaller models. two-stage fine-tuning also provides consistent performance improvements across model sizes. in contrast, data augmentation offers only marginal benefits, limited to the smaller 600m model, and the improvements appear inconsistent. these findings highlight the promise of scalable fine-tuning techniques for translation in lowresource settings. however, our focus on mt finetuning overlooks the more significant impact of asr errors, which remain a primary source of performance degradation in the end-to-end pipeline. this suggests that future research should prioritize improvements in the asr component. further work could also explore more targeted data augmentation strategies, end-to-end fine-tuning approaches, and incorporate qualitative evaluations with native speakers to better capture translation quality nuances.