The SETU - DCU Submissions to IWSLT 2024 Low-Resource Speech-to-Text Translation Tasks (2024)

Abstract: Natural Language Processing (NLP) research and development has experienced rapid progression in the recent times due to advances in deep learning. The introduction of pre-trained large language models (LLMs) is at the core of this transformation, significantly enhancing the performance of machine translation (MT) and speech technologies. This development has also led to fundamental changes in modern translation and speech tools and their methodologies. However, there remain challenges when extending this progress to underrepresented dialects and low-resource languages, primarily due to the need for more data. This paper details our submissions to the IWSLT speech translation (ST) tasks. We used the Whisper model for the automatic speech recognition (ASR) component. We then used mBART and NLLB as cascaded systems for utilising their MT capabilities. Our research primarily focused on exploring various dialects of low-resource languages and harnessing existing resources from linguistically related languages. We conducted our experiments for two morphologically diverse language pairs: Irish-to-English and Maltese-to-English. We used BLEU, chrF and COMET for evaluating our MT models.

## introduction the introduction of the transformer architecture @xcite is considered to be a groundbreaking development in nlp. this innovation has led to the rise of llms, which have become the catalyst for the ai revolution we are presently witnessing. llms have consistently pushed the boundaries of research by improving upon the state-ofthe-art across various nlp tasks. the variants of transformer such as the transducer @xcite , @xcite , and espnet @xcite have become essential to the success observed in a range of speech tasks, this study explores low-resource speech-totext translation, focusing on irish-to-english and maltese-to-english language pairs. we focused on developing our st systems following two standard approaches: the rest of the paper is organised as follows: section 2 describes our related work. our datasets are explained in section 3. section 4 describes the models we used. in section 5, we discuss our experiments and results. we conclude with avenues for future work in section 6. ## related work this section discusses some foremost papers related to our work. @xcite recently utilise llms for mt, such as mbart @xcite and @xcite , which they used within both e2e and cascaded st frameworks. furthermore, enhancements in asr were achieved by employing pseudo-labeling for data augmentation and adjusting for channel variations in telephone speech data. additionally, they employed minimum bayes-risk decoding to optimise the integration of their e2e and cascaded st systems. the proposed framework led to impressive results. @xcite utilised the fairseq s2t framework foot_0 , where they used log mel-scale filter bank @xcite features for audio representation and transformer for translation. their systems integrated asr and mt into the framework sequentially. the system's asr component was powered by a pre-trained xls-r model @xcite , enhanced with a fine-tuning step. at the same time, translations were performed using an mt system developed from a fine-tuned llm. they found that in low-resource settings, like quechua-to-spanish, direct st methods (combining asr and mt) tended to outperform standalone llm applications. mbuya and anastasopoulos (2023) used selfsupervised pre-trained speech models to improve translation performance in specific applications. their study utilised self-supervised models such as wav2vec 2.0 @xcite , xlsr-53, and hubert @xcite . their findings indicated that the wav2vec 2.0 and hubert models achieved similar performance levels in tasks involving low-resource languages and dialects. moreover, they found that the wav2vec 2.0 model performed better after removing its top three layers, a modification that the hubert model did not require. in contrast, the xlsr-53 model showed weaker results in low-resource contexts but excelled in translating dialects, outperforming both wav2vec 2.0 and hubert in those scenarios. @xcite investigated a novel approach termed "style embedding intervention" for low-resource formality control in spoken language translation. by assigning distinct style vectors to individual input tokens their proposed method comprehended and managed the subtleties of translating between formal and informal styles. they found that their approach surpasses previous "additive style intervention" techniques, particularly for the english-to-korean translation task, enhancing average matched accuracy. after analysing their "style embedding intervention" model, they found that most of the style information was acquired in @xmath0, further improving the average matched accuracy. ## cascaded system this section describes the architecture of our cascaded system. like standard cascaded st systems, our asr and mt models are interconnected, i.e. the output from the asr model serves as the input to the mt system. for this experiment, we selected the openai whisper model foot_2 @xcite as our asr system. we fine-tuned the openai whisper small model on maltese speech to optimise its asr capabilities. as for the mt component, we used two different pre-trained models, mbart-50 foot_3 @xcite and nllb-200distilled-600m foot_4 @xcite . both models were fine-tuned on the maltese-to-english bilingual data. as pointed out above, we used the openai whisper model as our asr system. we aligned the data format with the model's input requirements to prepare our data. this involved removing unnecessary data chunks from the dataset, eliminating special characters, and converting the sentences to lowercase. since our input audio was sampled at 48khz, we downsampled it to 16khz before passing it to the openai whisper feature extractor, as 16khz is the sampling rate expected by the model. additionally, we adjusted the audio inputs to the correct sampling rate using the "cast column" method. this operation does not alter the audio files directly but instead instructs the dataset to resample the audio samples on-the-fly the first time they are loaded. we empirically identified that the following hyperparameter settings provided us the best results: batch size of 16, learning rate of 1e-5, 500 warmup steps, 30,000 max steps, per-device eval batch size of 8, generation max length of 225, and intervals of 1,000 steps for saving and evaluating, and 25 steps for logging. ## end-to-end system our submission for the english-irish language pair comprises a fine-tuned version of openai whisper small foot_5 to perform direct st. note that in this experiment, the audio files are resampled at 16khz. this experiments were carried out for irish-to-english only. in terms of hyperparameters selection, for our e2e experimentation, we identified that the following settings provided us the best results: batch size of 16, learning rate of 1e-5, 500 warmup steps, 1 gradient accumulation steps, generation max length of 225, and intervals of 500 steps for saving and evaluating. the model was fine-tuned over three epochs. we also integrated early stopping with @xmath1. the data preprocessing pipeline was the same as the one used for the cascaded system (see section 4). ## results this section discusses the results that we obtained from our experiments. a cascaded system; however, this time, we used mbart-50 as the decoder. this setup provided us 44.70 bleu, 65.53 chrf, 0.796 comet points on the evaluation test set. fine-tuning openai's whisper model for the english-to-irish language pair has led to a performance improvement, despite the fact that the language is unsupported and unavailable in the model's training data. unfortunately, the bleu score remains low, probably due to instances of overgeneration and undergeneration. the chrf score, which measures character-based similarity, is higher but still indicates room for improvement as far as translation quality is concerned. as shown in ## conclusion in this study, we discussed our st models for the iwslt 2024 low-resource task for both irish-english and maltese-english language pairs. our proposed architecture offers numerous benefits: it is both computationally and data-efficient, supports both speech-to-text and text-to-text translations (including transcription), enhances knowledge transfer which boosts performance in low-resource languages, and exhibits robust translation capabilities. future investigations will focus on a detailed assessment of our architecture's asr functionality and explore the use of adapters within the speech representation model. additionally, a thorough examination of the optimal layers will be necessary when the speech representation model is not updated.