FINDINGS OF THE IWSLT 2021 EVALUATION CAMPAIGN (2021)

Abstract: The evaluation campaign of the International Conference on Spoken Language Translation (IWSLT 2021) featured this year four shared tasks: (i) Simultaneous speech translation, (ii) Offline speech translation, (iii) Multilingual speech translation, (iv) Low-resource speech translation. A total of 22 teams participated in at least one of the tasks. This paper describes each shared task, data and evaluation metrics, and reports results of the received submissions.

## introduction the international conference on spoken language translation (iwslt) is the premier annual scientific conference for all aspects of spoken language translation. for 18 years running @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite , the conference organizes and sponsors open evaluation campaigns around key challenges in simultaneous and consecutive translation, under real-time/low latency or offline conditions and under low-resource or multilingual constraints. system descriptions and results from participants' systems and scientific papers related to key algorithmic advances and best practice are published in proceedings and presented at the conference. iwslt is also the venue of the sigslt, the special interest group on spoken language translation of acl, isca and elra. with its long track record, iwslt benchmarks and proceedings serve as reference for all researchers and practitioners working on speech translation and related fields. this paper reports on the evaluation campaign organized by iwslt 2021, which features four shared tasks: on the use of multiple languages to improve supervised and zero-shot speech translation between four romance languages and english; the shared tasks were attended by 22 participants (see ## simultaneous speech translation simultaneous translation is the task of translating incrementally with partial text or speech input only. such capability enables multilingual live communication and access to multilingual multimedia content in real-time. the goal of this challenge, organized for the second consecutive year, is to examine systems that translate text or audio in a source language into text in a target language from the perspective of both translation quality and latency. ## offline speech translation offline speech translation, declined in various forms over the years, is one of the speech tasks with the longest tradition at the iwslt campaign. like in the last two evaluation rounds, this year foot_4 it focused on the translation of english audio data extracted from ted talks foot_5 into german. ## multilingual speech translation while multilingual translation is an established task, until recently, few parallel resources existed for speech translation and most remain only for translation from english speech. multilingual models enable transfer from related tasks, which is particularly important for low-resource languages; however, parallel data between two otherwise high-resource languages can also often be rare, making multilingual and zero-shot translation important for many resource settings. in addition to parallel speech and translations, many sources of data may be useful for speech translation: monolingual speech and transcripts, parallel text, and data from other languages or language pairs. while cascades of separately trained automatic speech recognition (asr) and machine translation (mt) models can leverage all of these data sources, how to most effectively do so with end-to-end models remains an open and exciting research question. ## low-resource speech translation the goal of the low-resource speech translation task is to investigate pathways for developing speech translation systems for currently underserved languages. the majority of the world's languages are predominantly oral, hence the need for speech-based language tools (translation included) is paramount for them to be of any use to the language community. at the same time, most of these languages are also under-resourced, with little to no data being available for speech transcription and translation. while offline speech translation has a longstanding tradition at the iwslt campaign and both monolingual and multilingual models offer impressive promises for downstream model deployment, the majority of recent advances in speech translation both require large amounts of data and are typically benchmarked on language pairs with such data abundance. however, for the vast majority of the world's languages there exist little speech-translation parallel data at the scale needed to train modern speech translation models. instead, in a real-world situation one will have access to limited, disparate resources (e.g. wordlevel translations, speech recognition, small parallel text data, monolingual text, raw audio, etc). the low-resource track aims to fill this gap, by encouraging and facilitating research on speechtranslation for data-scarce language pairs.