S peech T : Findings of the First Mentorship in Speech Translation (2025)

Abstract: This work presents the details and findings of the first mentorship in speech translation (SpeechT), which took place in December 2024 and January 2025. To fulfil the mentorship requirements, the participants engaged in key activities, including data preparation, modelling, and advanced research. The participants explored data augmentation techniques and compared end-to-end and cascaded speech translation systems. The projects covered various languages other than English, including Arabic, Bengali, Galician, Indonesian, Japanese, and Spanish.

## introduction at the beginning of the mentorship on speech translation, the participants were provided with the following descriptions and guidelines for each task: data: define, collect, and process bilingual speech data in a chosen language. your dataset should consist of "train", "dev/validation", and "test" splits. by the end of the task, each participant should share a hugging face link to their datasets. the dataset page metadata should include sections for data sources, processing steps you applied in detail, and credits/citations of the original datasets. modelling: choose one of the popular models, e.g. whisper @xcite or wav2vec @xcite , and fine-tune it on the data prepared in the first task. experimenting with different fine-tuning approaches and hyperparameters is encouraged. by the end of the task, the participants should share their fine-tuned models, and evaluation scores on the test dataset. advanced research: enhance the quality of your model through experimenting with advanced approaches, including creating synthetic data @xcite , comparing end-to-end ⋆ correspondence: yasmin[at]machinetranslation.io * participant in the mentorship systems to cascaded systems @xcite , using language models (e.g. n-grams) @xcite , domain adaptation @xcite , or any other valid approach. by the end of the task, the participants should share their advanced models. they should also clarify how the advanced approach improved the speech translation quality compared to the original fine-tuned model. release & publication: write the project details to publish as a paper. moreover, the outcomes of all the projects are publicly accessible. foot_0 2 end-to-end vs. cascaded systems speech translation systems can be (a) "cascaded" systems, or (b) "end-to-end" systems @xcite . cascaded speech translation systems use two models, one for auto-matic speech recognition (asr) and one for textual machine translation (mt) (cf. figure 1 ). end-toend speech translation systems use one model for the whole process (cf. figure 2 ). ## approaches to synthetic data when the data is limited for the language or domain, synthetic data can be used to augment the authentic data. synthetic data for speech translation systems can be generated in diverse methods, including: ## projects most of the projects used a mix of data augmentation of authentic data with synthetic data, finetuning models, and comparing the performance of "end-to-end" speech systems to "cascaded" systems (cf. section 2). participants used the hugging face transformers library to fine-tune pretrained models. they fine-tuned whisper @xcite for "endto-end" speech translation, and for asr in the "cascaded" system. moreover, they fine-tuned @xcite ) for text-to-text translation as part of "cascaded" speech translation systems. for evaluation, they used the sacrebleu library @xcite to obtain bleu @xcite and chrf++ @xcite scores. in addition, one of the participants calculated comet scores @xcite . for inference, they either used the transformers library or faster-whisper (based on ctranslate2 @xcite ) for audio translation and transcription with whisper. for text-to-text translation with opus and nllb-200 models, some of them used the transformer library directly while others used ctranslate2 with float16 quantization, which is more efficient. for synthetic data generation, they used chatgpt (ope-nai, 2023) and opus (tiedemann and thottingal, 2020) models. given that each participant chose a language pair, we dedicate a section for each project based on the language pair, including galicianto-english, indonesian-to-english, spanish-to-japanese, arabic-to-english, and bengali-to-english. each language section describes data, modelling, and evaluation of each project. ## conclusions the speecht mentorship brought together several practitioners and students from diverse companies and institutions across the world to explore speech translation. the participants have diverse backgrounds, ranging from generic software knowledge to text-to-text mt experience. ultimately, five participants have made successful submissions and contributed to this work (cf. section 6).