JHU IWSLT 2023 Dialect Speech Translation System Description (2023)

Abstract: This paper presents JHU’s submissions to the IWSLT 2023 dialectal and low-resource track of Tunisian Arabic to English speech translation. The Tunisian dialect lacks formal orthography and abundant training data, making it challenging to develop effective speech translation (ST) systems. To address these challenges, we explore the integration of large pre-trained machine translation (MT) models, such as mBART and NLLB-200 in both end-to-end (E2E) and cascaded speech translation (ST) systems. We also improve the performance of automatic speech recognition (ASR) through the use of pseudo-labeling data augmentation and channel matching on telephone data. Finally, we combine our E2E and cascaded ST systems with Minimum Bayes-Risk decoding. Our combined system achieves a BLEU score of 21.6 and 19.1 on test2 and test3, respectively.

## introduction the performance of machine translation systems is closely tied to the amount of available training data. regional dialects, which are less prevalent and primarily spoken languages, pose a challenge for these systems due to the scarcity of digital data, the absence of standard orthography, and prevalence of non-standard grammar. the iwslt 2023 dialect and low-resource track focuses these challenges. in this paper we present the jhu tunisian arabic to english speech translation systems submitted to the iwslt 2023 dialectal and low-resource track @xcite . arabic and its dialects form a dialect continuum anchored by modern standard arabic (msa) @xcite . while msa is the language of formal and written communication, most native arabic speakers colloquially use local dialects, which often lack a standardized written form. in many north african arabic dialects, including tunisian, there is a significant code-switching with and borrowing from several contact languages: berber and romance languages like french, spanish and italian. recent successes in machine translation (mt) of text for low-resource languages or non-standard dialects have entailed the use of large pretrained models such as mbart @xcite ) and nllb @xcite . these models have demonstrated state-of-the-art performance via transfer learning from higher-resource languages, particularly through related languages. however, there is a lack of understanding regarding how to effectively integrate these models with speech recognition systems to develop speech translation systems. to fill this gap we investigate dialect transfer by integrating large pretrained models with speech recognition models in end-to-end (e2e) and cascaded speech translation (st) systems. the key components of our system are: our system outperforms the best previous approaches @xcite for both asr (wer) and st (bleu). we also found that integrating pre-trained mt models into end-toend st systems did not improve performance. the dialect speech translation task permitted submissions using models trained under two data conditions, (a) constrained and (b) unconstrained. for brevity, we will refer to these conditions as (a) and (b) respectively. ## methods in this section we describe our cascaded ( §3.1), and end-to-end (e2e) ( §3.2) speech translation systems as well as our strategy for combining both approaches ( §3.3). ## experiments in this section, we describe our experiments on the asr, mt, and st tasks. in order to reduce the orthographic variation in the tunisian speech transcription we performed additional text normalization similar to @xcite which showed significant improvements on asr, mt and st tasks. the normalization is performed on both tunisian and msa transcripts and includes removing diacritics and single character words, and alif/ya/ta-marbuta normalization (see @xcite for more details). ## conclusion in this paper, we have presented our submission for the iwslt 2023 dialect speech translation task. we compared end-to-end to cascaded systems under constrained and unconstrained conditions. we found that an e2e-st system outperformed the cascaded system under the constrained condition, while the cascaded models significantly outperformed the e2e-st systems under the unconstrained condition. we provided a new e2e-st baseline combining large pretrained mt with asr under the unconstrained condition. finally, we demonstrated that pseudo-labeling and channel matching provided significant improvements for the asr and hence improved cascaded st systems. in future work we plan to explore more effective ways of integrating the large pretrained mt models into e2e st systems.