HW - TSC ’s Submissions To the IWSLT 2024 Low-resource Speech Translation Tasks (2024)

Abstract: In this work, we submitted our systems to the low-resource track of the IWSLT 2024 Speech Translation Campaign. Our systems tackled the unconstrained condition of the Dialectal Arabic North Levantine (ISO-3 code: apc) to English language pair. We proposed a cascaded solution consisting of an automatic speech recognition (ASR) model and a machine translation (MT) model. It was noted that the ASR model employed the pre-trained Whisper-large-v3 model to process the speech data, while the MT model adopted the Transformer architecture. To improve the quality of the MT model, it was stated that our system utilized not only the data provided by the competition but also an additional 54 million parallel sentences. Ultimately, we reported that our final system achieved a BLEU score of 24.7 for apc-to-English translation.

## introduction the iwslt 2024 speech translation campaign featured a low-resource track that posed a challenging task: translating dialectal arabic speech to english text. this language pair is particularly demanding due to the scarcity of available training data and the complexity of handling dialectal arabic variations. to tackle this problem, we propose a cascaded approach that leverages state-of-the-art models for automatic speech recognition (asr) and machine translation (mt). end-to-end models, which directly map audio inputs to translated text outputs, heavily rely on the availability of paired audio and transcription data. for low-resource tasks, acquiring such data can be exceptionally challenging. consequently, we adopted a cascaded model architecture, which decouples the speech recognition and translation components. this approach allows us to leverage additional parallel text data to enhance the mt module's performance, ultimately benefiting the overall speech translation task. our asr model is built upon the whisper-large-v3 architecture, a powerful pre-trained model that has demonstrated impressive performance in transcribing diverse speech data.by employing this model, we aim to accurately transcribe the dialectal arabic speech inputs, to capture the nuances and variations present in the spoken language. for the mt component, we adopt the transformer architecture @xcite , which has become the de facto standard for modern neural machine translation systems. the transformer model is known for its ability to effectively capture long-range dependencies and produce high-quality translations, making it well-suited for the task at hand. to further enhance the performance of our mt system, we augment the provided training data with a substantial amount of additional parallel data, totaling 54 million sentence pairs. this data augmentation strategy aims to improve the model's robustness and generalization capabilities, enabling it to better handle the complexities of translating between dialectal arabic and english. by combining the strengths of these two powerful models in a cascaded fashion, we aim to deliver a robust and accurate speech-to-text translation system for the iwslt 2024 low-resource track, pushing the boundaries of what is achievable in this challenging language pair. ## methods we employed a cascade approach for the spoken language translation (st) task, leveraging both automatic speech recognition (asr) and data source volume iwslt 2024 official dataset 27m additional arabic-english bilingual data 54m total 81m machine translation (mt) models. the cascade model consists of two stages: the asr stage and the mt stage. ## results the multi-step fine-tuning method first pretrains a base model on large-scale general-domain corpora, and then conducts multiple rounds of fine-tuning on the task-specific data, with each round optimizing the model based on the previous round. this approach leverages general knowledge, addresses data distribution mismatch issues, and avoids overfitting. after each round of fine-tuning, the bleu metric is used to evaluate the translation quality, serving as the basis for determining whether to proceed with the next round of fine-tuning. through this gradual fine-tuning process, the model's performance can be progressively enhanced. ## conclusion our research has led to the following key conclusions: firstly, for the arabic-to-english translation task, incorporating additional bilingual corpus data significantly enhanced model performance. these corpora contained rich knowledge of arabic dialects, enabling the model to better learn and translate these special language variants, thereby improving the overall translation quality. secondly, we adopted advanced training strategies such as curriculum learning and r-drop, which also brought substantial performance gains to the machine translation model. curriculum learning facilitated gradual learning from easy to difficult scenarios, while r-drop effectively mitigated overfitting issues and improved the model's generalization capability. these strategies were the core methods employed in our submission, yielding outstanding practical results.