The KIT Speech Translation Systems for IWSLT 2024 Dialectal and Low-resource Track (2024)

Abstract: This paper presents KIT’s submissions to the IWSLT 2024 dialectal and low-resource track. In this work, we build systems for translating into English from speech in Maltese, Bemba, and two Arabic dialects Tunisian and North Levantine. Under the unconstrained condition, we leverage the pre-trained multilingual models by fine-tuning them for the target language pairs to address data scarcity problems in this track. We build cascaded and end-to-end speech translation systems for different language pairs and show the cascaded system brings slightly better overall performance. Besides, we find utilizing additional data resources boosts speech recognition performance but slightly harms machine translation performance in cascaded systems. Lastly, we show that Minimum Bayes Risk is effective in improving speech translation performance by combining the cascaded and end-to-end systems, bringing a consistent improvement of around 1 BLUE point.

## introduction in this paper, we describe kit's systems submitted to iwslt 2024 dialectal and low-resource track. we focus on three language pairs: bemba (iso code: bem) to english, maltese (iso code: mlt) to english, and dialectal arabic to english. the dialectal arabic language pair evaluates the performance of two arabic vernaculars, namely tunisian (iso code: aeb) and north levantine (iso-3 code: apc). maltese and tunisian language pairs are available in iwslt2023 @xcite , and the others are newly included this year. the submissions are under unconstrained conditions to leverage pre-trained models and additional data resources. recent advancements in dialectal and lowresource speech translation show the benefits of utilizing pre-trained models @xcite @xcite . nowadays, the capacities of pre-trained models are expanded by incorporating more extensive data and expanding language coverage. this work leverages the state-of-the-art pretrained models, including seamelssm4t @xcite , @xcite , and nllb @xcite . cascaded and end-to-end (e2e) are popular speech translation (st) systems. the cascaded system consists of automatic speech recognition (asr) and machine translation (mt) models, while the e2e systems integrate both functions into one model. recent work shows the e2e system shows comparable performance to the cascaded system in speech translation @xcite @xcite , while there needs research to show which system performs better on dialectal and lowresource scenarios @xcite @xcite @xcite . building st systems for low-resource datasets always suffers from data limitations. accordingly, we collect available training resources and investigate the training strategies for using them. although datasets other than the development data might introduce domain differences that could potentially model performance, we explore the benefits of using extra-supervised data. furthermore, we investigate adapter fine-tuning training to address data scarcity. by freezing the pre-trained parameters and only fine-tuning the adapter parameters, this approach decreases the number of trainable parameters. in addition to building st systems, this work explores the decoding approach minimum bayes risk(mbr) to re-rank the candidate translation @xcite from the built systems. we explore the combination of individual systems and across systems, and our findings suggest combining translations from the cascaded and the e2e systems is effective for all language pairs. 2 data description ## method we conduct preliminary evaluations on tunisian dialects to assess systems performance and then apply the promising approaches to other languages for effective analysis. the motivation is that the tunisian language pair has effective systems from iwslt 2023 @xcite ) for approach analysis.