Recent Highlights in Multilingual and Multimodal Speech Translation (2024)

Abstract: Speech translation has witnessed significant progress driven by advancements in modeling techniques and the growing availability of training data. In this paper, we highlight recent advances in two ongoing research directions in ST: scaling the models to 1) many translation directions (multilingual ST) and 2) beyond the text output modality (multimodal ST). We structure this review by examining the sequential stages of a model’s development lifecycle: determining training resources, selecting model architecture, training procedures, evaluation metrics, and deployment considerations. We aim to highlight recent developments in each stage, with a particular focus on model architectures (dedicated speech translation models and LLM-based general-purpose model) and training procedures (task-specific vs. task-invariant approaches). Based on the reviewed advancements, we identify and discuss ongoing challenges within the field of speech translation.

## introduction speech translation (st) is the task of automatically converting speech in a source language into its equivalent in a target language. recently, there has been significant interest in multilingual models @xcite @xcite @xcite that serve a broad range of translation directions, as well as multimodal models @xcite @xcite that not only generate text translations but can also synthesize speech output. 1 both developments are crucial steps towards making st technologies more inclusive. by expanding language coverage and offering diverse output modalities, these advancements make st models accessible 1 here we restrict our discussion to the two modalities of speech and text. we acknowledge the relevance of additional modalities, such as vision, and leave them for open questions. to a wider range of users, allowing them to interact with the technology in their preferred language and format. besides the practical relevance, multilingual and multimodal translation are instances of multi-task learning @xcite , a central machine learning challenge. in this paper, we aim to review recent advancements in multilingual and multimodal st. we structure the review by the stages in a model's development lifecycle, as illustrated in figure 1 . these stages consist of model coverage and architecture selection, training procedures, evaluation methodologies, and deployment considerations. in the review of current model architectures ( §3), besides discussing dedicated models for translation, we review emerging models in adapting text-based large language models (llms) for speech processing. given the inherent multi-task learning nature of both multilingual and multimodal st, we put special emphasis on the learning procedure ( §4). specifically, we take two perspectives from taskspecific and task-invariant modeling, and discuss their roles in terms of the trade-off between interference and transfer. while prioritizing direct st, we also review related multilingual and multimodal techniques in automatic speech recognition (asr) and text-totext machine translation (mt), as they often are extendable to st tasks. we also note that this work is not an exhaustive survey, but rather aims to highlight directions of recent developments and provide context for open challenges. ## training resources determining training resources is one of the initial steps when building a speech translation model. this section provides a brief overview of the language and modality coverage ( §2.1) in existing training resources, followed by discussions on scaling datasets by augmentation or mining ( §2.2). training resources model architecture multilingual and multimodal learning evaluation deployment language & modality coverage data augmentation & mining dedicated s2t models dedicated s2s models general-purpose models speech discretization task-specific modeling mixture-of-expert (moes) adapters factorization pruning task-invariant modeling evaluation resource evaluation metric compression and distillation continual learning inference-time customization dataset directions modality & type # lang. pairs total hours must-c (di gangi et al., 2019; cattoni et al., 2021) @xmath0.4k europarl-st (iranzo-sánchez et al., 2020) @xmath1.5k covost 2 @xcite @xmath2, @xmath3.4k voxpopuli @xcite @xmath4, interpretation 210 17k cvss @xcite @xmath5, synthesized 21 2k speechmatrix @xcite @xmath6, mined 136 418k ## model architecture in this section, we first review dedicated model architectures for speech-to-text (s2t; §3.1) and speech-to-speech (s2s; §3.2) translation, with a focus on the use of foundation models. afterwards, we discuss recent developments in adapting general-purpose llms ( §3.3) for encoding or generating speech. ## multilingual and multimodal learning both multilingual and multimodal speech translation are instances of multi-task learning, where each translation direction in one input-output modality pair corresponds to one task. as also observed in general multi-task learning @xcite , a key goal here is to maximize the transfer while minimizing the interference between tasks, while maintaining an efficient trade-off @xcite . given a defined model architecture ( §3), different training procedures control the learned representations. in this section, we will discuss the relevant approaches in detail, taking two perspectives from task-specific ( §4.1) and taskinvariant modeling ( §4.2). ## evaluation the evaluation of multilingual and multimodal st models relies on more resources than their bilingual and unimodal counterparts. here we outline relevant developments in evaluation resources ( §5.1) and metrics ( §5.2). ## deployment in this section, we review three aspects relevant to model deployment: compression and distillation for serving the models ( §6.1), continual learning of new capabilities ( §6.2), and inference-time customization ( §6.3). ## conclusion in this paper, we presented a selection of recent advancements in multilingual and multimodal speech translation. we zoom into individual stages of the lifecycle of building a system: from determining model coverage and architecture, training procedures, to evaluation, and eventually deployment. this work is not an exhaustive survey, but rather a snapshot of ongoing developments related to multilingual and multimodal speech translation. we welcome the community's feedback on any relevant omitted works in the current version.