U o M - DFKI submission to the low resource shared task (2024)

Abstract: This system description paper presents the details of our primary and contrastive approaches to translating Maltese into English for IWSLT 24. The Maltese language shares a large vocabulary with Arabic and Italian languages, thus making it an ideal candidate to test the cross-lingual capabilities of recent state-of-the-art models. We experiment with two end-to-end approaches for our submissions: the Whisper and wav2vec 2.0 models. Our primary system gets a BLEU score of 35.1 on the combined data, whereas our contrastive approach gets 18.5. We also provide a manual analysis of our contrastive approach to identify some pitfalls that may have caused this difference.

## introduction in this paper, we describe the uom-dfki submission to the dialectical and low-resource track of the iwslt 2023 evaluation campaign, focusing on the unconstrained approach for the maltese to english track. maltese is considered a hybrid language, with most vocabulary coming from arabic, italian, and english. while there is a major overlap with arabic, maltese data uses latin instead of arabic script. our main focus for this submission is using publicly available multilingual models to exploit the multilingual capabilities of models, given the interesting mixture of vocabulary in the maltese language. for this paper, we focus on end-to-end approaches for spoken language translation (slt), namely with whisper @xcite and wav2vec 2.0 xls-r @xcite . we use the whisper-based model as our primary submission and the wav2vec 2.0 model as the contrastive approach. the whisper system is pre-trained on 680,000 hours of speech data using an encoder-decoder method. a substantial amount of the training data, nearly one-fifth, is english audio, and 9,000 hours is maltese. @xcite claim that with 41 hours of maltese translation data, the whisper model is able to achieve roughly 14 bleu points. in this paper, we use the data released for this task to fine-tune the whisper model further. there are various whisper models with varying parameter sizes. @xcite shows how, with larger parameters, the whisper architecture performs better in the asr setting for maltese asr. for this work, we decided to use the most recent whisper model; the largest model is whisper-large-v3. our approach for wav2vec 2.0-based models also consisted of using an encoder-decoder approach, namely speechen-coderdecoder framework @xcite , as made available on huggingface @xcite . we worked with three different models as our decoder for our contrastive approaches, namely bert @xcite and mbart fine-tuned for machine translation from different languages into english @xcite . ## literature review the iwslt low-resource and dialectical shared task increased the number of language pairs they released data for in 2023. in the 2022 edition of the workshop, @xcite released the data for teams to develop systems to transcribe and translate the low-resource language pairs of tamasheq-english and tunisian arabic-french. in the 2023 edition of the task, however, @xcite extended the task to include the language pairs irish-english, maltese-english, pashto-french and quechua-spanish. in 2022, three teams submitted models for tamasheq-english: on-trac @xcite , taltech and gmu. on-trac also submitted to the tunisian arabic-french pair, like cmu @xcite and jhu @xcite did. in 2023, gmu submitted models for irish-english, marathi-hindi, pashto-french and tamasheq-french (mbuya and anastasopoulos, 2023), alexa ai submitted models for marathi-hindi and tamasheq-french (shanbhogue et al., 2023), on-trac submitted for tamasheq-french and pashto-french @xcite , naver submitted for tamasheq-french and quechua-spanish @xcite , but @xcite and @xcite submitted only for marathi-hindi, quespa submited for quechua-spanish (e. @xcite and um-dfki submitted for maltese-english @xcite . these teams employed various techniques, ranging from traditional cascade systems to various end-to-end architectures. many teams leveraged large pre-trained models, including xls-r, mbart, wav2vec 2.0 and hubert. the alexa ai team tried an interesting approach by focusing on data augmentation, ensemble modelling and post-processing techniques to improve their results. transformer models for mt were popular across the board. the naver submissions obtained particularly good results in their respective language pairs by using pre-trained asr and mt models from the nllb project @xcite , which include both tamasheq and quechua in their training, showing the importance of language diversity in multilingual models. ## dataset in this section, we briefly describe the dataset used to fine-tune our systems. we include a description of the dataset used to fine-tune the mbart50 many-to-one model @xcite and the dataset released for this shared task. mbart50 many-to-one @xcite utilized and released the ml50 dataset for fine-tuning the mbart model for translating in 50 languages. it uses english as a pivot language to collect parallel data for 49 other languages from sources such as iwslt, ted, wat, etc. it is also noted that the 49 languages selected for the dataset are based on language family, available mono-lingual data and parallel data. this, in turn, means that the dataset is not balanced and results in better performance improvements for high-resource languages compared to low-resource languages. we used the data released for this shared task to fine-tune our models. namely, the two training sets created from the common voice and masri maltese speech corpora. subsets from these larger corpora were extracted, 5 hours and 11 minutes from the verified common voice data and 6 hours and 39 minutes from the masri-headset corpus. the transcription of each sample was translated. fine-tuning whisper for speech translation requires audio for input and the transcription of that audio in sentence form as a target. we pre-processed the input text so that numbers were written in words and no punctuation or capitalization was included. given how they were acquired, we note the difference between the subset released from the masri corpus and the commonvoice dataset. while the masri corpus provides clean and nearly noise-free audio samples, commonvoice samples vary in terms of different noises and the quality of audio-capturing devices. ## experiments in this section, we briefly describe different experiments we conducted for our submissions, including those we did not submit for evaluation. first, we describe our experiments with the wav2vec2-xls-r @xcite model, followed by the whisperbased @xcite models. we utilized huggingface @xcite ) libraries for our experiments. ## results & discussion in this section, we discuss the results from both submissions. as per the participation instruction, the results are reported individually for the com-monvoice subset, masri subset and the combined testset. a rudimentary manual analysis of our constrained system shows a common theme of repeated phrases across some bad translations. for example, for the file msrts_m_03_ts_00016.wav, our contrastive system produced "our words are not 'as it were', the people's words are not 'as they should be', our words are not 'as they should be', our words are not 'as they should be'", whereas for the file msrts_m_09_ts_00008.wav, it produced "and he comes running" repeated 8 times. we did not find any conclusive pattern of this repetition based on the output text length, as in some instances, we find that only a sub-phrase is repeated one or more times towards the end of the output. we experimented with different output token lengths while debugging this behaviour, but it did not yield any conclusive reason, as it was present while using different inference strategies as well. another approach to fix this behaviour would be a post-fix approach where we automatically fix the output with repeated substring search. in this study, we did not utilize such an approach and left it for future work. we analyzed the performance of our primary submission method using speech in a code-switched conversation (hindi and english) and found whisper auto-translating the hindi part to english in a few instances when we put the input language as "en". the nature of these fixes is not deterministic in this preliminary experiment, as we saw different segments translated in different runs. however, due to the end-to-end nature of our approaches, we are uncertain if this is the case with our model as well. we attribute the improved performance of the whisper model to the increased pre-training of the model on more data than wav2vec2-xls-r. however, without inspection of the data and the high domain sensitivity in maltese, it remains difficult to quantify the effect. we also note that our models' performance on the testset closely resembles the results we obtained on the dev set during our training. our primary model scored 35.9 on the dev set, whereas it scored 35.1 on the testset; similarly, our contrastive model scored 18.5 on both dev and test splits. ## conclusion & future work in our end-to-end translation system experiments, we report that the whisper-based model outperforms our speechencoderdecoder model. the performance of our contrastive model is much worse for the masri subset than that of the common-voice subset. we report that multi-lingual pretraining and fine-tuning can provide good-quality translation output in an end-to-end approach. we also report that since whisper is already trained in a semi-supervised manner, the model output had to be re-processed to produce the ideal results for this work. overall, the results are much better compared to @xcite for iwslt 2023. however, we note that in the previous edition, the test dataset drastically differed in quality and domain compared to the test set for this year's shared task. however, it is not possible to draw a parallel for this comparison as the test set in the iwslt'23 edition consisted of a podcast episode, which is more colloquial in nature. it also suffered from poor asr outputs as there were instances of speakers talking over each other. another hypothesis is that while much of the training data for the mt part of the previous submission contains legal domain data, which has more influence from italian, the colloquial speak has more influence from arabic. based on the performance of our speechen-coderdecoder model, we hypothesize that data augmentation and combining parallel data from arabic and italian may improve the models' performance. we aim to extend this study with an analysis of gain/drop in performance when using a fine-tuned mbart50 as a translation system from arabic and italian to english, compared to using the same model as the decoder in this encoderdecoder setting. we also aim to investigate the auto-translation capabilities of whisper-based models by using them in a pipeline-based approach as well. furthermore, using language-specific adapters to leverage models trained only on asr or nmt data enables slt in low-resource contexts. previous work on this area @xcite , @xcite , including previous submissions to iwlst @xcite achieved high bleu scores and found that this method works particularly well in low-resource contexts. we also aim to explore this approach in the future with related languages such as italic and arabic, as it shows promise for maltese-english slt.