JHU IWSLT 2024 Dialectal and Low-resource System Description (2024)

Abstract: Johns Hopkins University (JHU) submitted systems for all eight language pairs in the 2024 Low-Resource Language Track. The main effort of this work revolves around fine-tuning large and publicly available models in three proposed systems: i) end-to-end speech translation (ST) fine-tuning of Seamless4MT v2; ii) ST fine-tuning of Whisper; iii) a cascaded system involving automatic speech recognition with fine-tuned Whisper and machine translation with NLLB. On top of systems above, we conduct a comparative analysis on different training paradigms, such as intra-distillation for NLLB as well as joint training and curriculum learning for SeamlessM4T v2. Our results show that the best-performing approach differs by language pairs, but that i) fine-tuned SeamlessM4T v2 tends to perform best for source languages on which it was pre-trained, ii) multi-task training helps Whisper fine-tuning, iii) cascaded systems with Whisper and NLLB tend to outperform Whisper alone, and iv) intra-distillation helps NLLB fine-tuning.

## introduction with recent developments in data-driven machine learning and transformer-based models @xcite , speech translation (st) systems (which accept spoken input in one language and automatically output corresponding text in another) have undergone major strides in performance @xcite @xcite . while these works demonstrate the effectiveness of using large pretrained models for speech translation between high-resource language pairs and establish new state-of-the-art (sota) performance in these setups, less attention has been devoted to whether these advances also benefit lowresource language pairs, and how they compare with sota systems for these languages. some of the populations with the greatest need for st tools are those speaking low-resource languages, which typically have less institutional support and funding for the development for nlp and speech tools @xcite @xcite : some speak minority languages in the areas where they live and need translation tools to communicate across a language barrier, or to consume or search for information more effectively online @xcite . certain populations speaking low-resource languages may also have low literacy rates or limited writing traditions in their native languages, increasing the imperative for speech-based, rather than text-based, translation systems @xcite . in this work, we developed st systems for eight language pairs, as organized in the iwslt 2024 dialectal and low-resource speech translation shared task. we approached this problem by leveraging systems pre-trained on a large amount of multilingual data and subsequently fine-tuning them for specific tasks: both end-to-end speech st and cascaded st (i.e. transcription followed by text-based translation). we compared different approaches and pre-trained models for each language pair, and we experimented with combining data from multi-ple related languages into the same train set. among the systems introduced, the approaches based on seamlessm4t v2 @xcite outperform others for language pairs that it has seen during pretraining and for which supervised st data are available (e.g. mar-hin, gle-eng, bho-hin, and mlt-eng). in other cases, a cascaded system is the most successful of the proposed approaches, namely, for apc-eng, bem-eng, que-spa, and tmh-fra. ## task description on the challenge website this year, foot_0 the organizers stated, "the goal of this shared task is to benchmark and promote speech translation technology for a diverse range of dialects and low-resource languages." to forward this aim, this year's task focuses on st for eight language pairs: levantine arabic to english (apc-eng), bemba to english (bem-eng), bhojpuri to hindi (bho-hin), irish to english (gle-eng), maltese to english (mlt-eng), marathi to hindi (mar-eng), quechua to spanish (que-spa), and tamasheq to french (tmh-fra). levantine is one of the most spoken arabic dialects, with the majority native-speaking populations in syria, lebanon, palestine, and jordan. both levantine arabic and maltese are semitic languages of the afroasiatic family. bemba is a bantu language of the niger-congo family, spoken by over 30% of zambia's population @xcite . bhojpuri, hindi, and marathi are indo-aryan languages; hindi and marathi are scheduled languages in india and have government backing for their support, whereas bhojpuri, like many other languages on the so-called hindi belt, lacks official status, has a much smaller writing tradition, and is only recently gaining attention in nlp @xcite @xcite . each of the source languages is low-resource, with tamasheq, bemba, and levantine arabic having the fewest wikipedia articles overall @xcite . despite their low digital support, these languages have a large native speaker base, including marathi's 83 million, according to ethnologue. 2 the organizers provide different varieties of data for each of these language pairs. we used predominantly provided datasets, along with some external data, all of which are outlined in though this year's task accepts both unconstrained submissions, allowing the use of external datasets and pre-trained models, and constrained submissions, our submission is limited to the unconstrained track, since all of our methods involved fine-tuning pre-trained models. ## proposed methods we introduce three primary frameworks, which are applied to different language pairs according to the availability of the data: (1) we fine-tune seam-lessm4t v2 for end-to-end st using e2e data; (2) we fine-tune whisper @xcite for endto-end st using e2e (and optionally asr) data; we explore various methodological additions to these methods. we look at joint fine-tuning and curriculum learning with the seamlessm4t v2-based approaches. we investigate several finetuning setups for the whisper-based systems, including pseudo-translation fine-tuning, multitask training with asr and mt as well as asr-only and st-only fine-tuning. we also looked at intradistillation as a method of enhancing nllb in mt. these ideas are further detailed below. @xcite introduce seamlessm4t v2, a model capable of end-to-end expressive and multilingual translations in a streaming fashion. seamlessm4t v2 supports multilingual input and output in both speech and text modalities, with a dedicated sub-model handling each modality combination. it has 2.3b parameters and is pretrained on 1m hours of unlabeled audio in 143 languages, using the w2v-bert xl architecture @xcite . it is then fine-tuned on text mt into english (x-eng) for 95 languages, asr for 96 languages, st into english for 89 languages, and speech-tospeech translation into english for 95 languages, and out of english eng-x for 35 languages. the pretraining languages of seamlessm4t v2 include english, irish, maltese, hindi, marathi, and arabic, foot_2 but not quechua, tamasheq, or bemba. ## results and discussion ## conclusion and future work in this work, we describe our submitted systems for all eight language pairs in the iwslt 2024 low-resource language track. we explore various fine-tuning approaches for large publicly available pre-trained models, compare end-to-end and cascaded systems, as well as investigate the benefits of joint and curriculum training, multitask learning, as well as intra-distillation. we find that the best-performing strategy is language-pair dependent, with fine-tuned seamlessm4t v2 generally performing best on languages that are included in its pretraining corpus. fine-tuned whisper generally performed better with multi-task fine-tuning than standard fine-tuning, and better still when employed in a cascaded system with fine-tuned nllb (with best results employing intra-distillation). for future improvements, augmenting mt finetuning data with asr hypotheses, as in @xcite , could equip nllb better for cascaded st. future work could also employ data augmentation of text and speech data, as in shanbhogue et al. (2023), via textual back-translation @xcite , speech synthesis for augmentation @xcite , or other methods. lastly, future research could employ the use of sslr, or employ the large amounts of raw the instances that are over 30 seconds when training seamlessm4t v2 and limit the generation length to 256 new tokens. 200 (a) mlt-eng training set (b) mlt-eng development set (c) mar-hin training set (d) mar-hin development set (e) que-spa training set (f) que-spa development set figure 3: length distribution (seconds) for each language pair (continued).