Findings of the IWSLT 2022 Evaluation Campaign (2022)

Abstract: The evaluation campaign of the 19th International Conference on Spoken Language Translation featured eight shared tasks: (i) Simultaneous speech translation, (ii) Offline speech translation, (iii) Speech to speech translation, (iv) Low-resource speech translation, (v) Multilingual speech translation, (vi) Dialect speech translation, (vii) Formality control for speech translation, (viii) Isometric speech translation. A total of 27 teams participated in at least one of the shared tasks. This paper details, for each shared task, the purpose of the task, the data that were released, the evaluation metrics that were applied, the submissions that were received and the results that were achieved.

## introduction the international conference on spoken language translation (iwslt) is the premier annual scientific conference for all aspects of spoken language translation. iwslt is organized by the spe-cial interest group on spoken language translation, which is supported by acl, isca and elra. like in all previous editions @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite , this year's conference was preceded by an evaluation campaign featuring shared tasks addressing scientific challenges in spoken language translation. this paper reports on the 2022 iwslt evaluation campaign, which offered eight shared tasks: the shared tasks attracted 27 participants (see ## simultaneous speech translation simultaneous translation is the task of generating translations incrementally given partial text or speech input only. such capability enables multilingual live communication and access to multilingual multimedia content in real time. the goal of this challenge, organized for the third consecutive year, is to examine systems that translate text or audio in a source language into text in a target language from the perspective of both translation quality and latency. ## offline speech translation offline speech translation, defined in various forms over the years, is one of the speech tasks with the longest tradition at the iwslt campaign. this year, foot_3 it focused on the translation of english audio data extracted from ted talks foot_4 into text in one of the three target languages comprising the 2022 sub-tasks, i.e. german, japanese, and mandarin chinese. ## speech to speech translation speech-to-speech translation is the task of translating audio input in a language into audio output in a target language. in the offline setting, systems are able to take into account an entire input audio segment in order to translate, similar to a consecutive interpreter. this is in contrast to streaming or simultaneous settings where systems are only exposed to partial input as in simultaneous interpretation. the goal of this task is to foster the development of automatic methods for offline speechto-speech translation. ## low-resource speech translation this shared task focuses on the problem of developing speech transcription and translation tools for under-resourced languages. for the vast majority of the world's languages there exist little speech-translation parallel data at the scale needed to train speech translation models. instead, in a real-world situation one might have access to limited, disparate resources (e.g. word-level translations, speech recognition, small parallel text data, monolingual text, raw audio, etc). building on last year's task that focused on two varieties of swahili @xcite , the shared task invited participants to build speech translation systems for translating out of two predominantly oral languages, tamasheq and tunisian arabic, and into the linguae francae of the respective regions (english and french). the use of any pre-trained machine translation, speech recognition, speech synthesis, or speech translation model was allowed, as did unconstrained submissions potentially using data other than the ones the organizers provided. ## dialect speech translation in some communities, two dialects of the same language are used by speakers under different settings. for example, in the arabic-speaking world, modern standard arabic (msa) is used as spoken and written language for formal communications (e.g., news broadcasts, official speeches, religion), whereas informal communication is carried out in local dialects such as egyptian, moroccan, and tunisian. this diglossia phenomenon poses unique challenges to speech translation. often only the "high" dialect for formal communication has sufficient training data for building strong asr and mt systems; the "low" dialect for informal communication may not even be commonly written. with this shared task (new for 2022), we hope to bring attention the unique challenges of dialects in diglossic scenarios. ## formality control for slt machine translation (mt) models typically return one single translation for each input segment. specific problems can arise for spoken language translation from english into languages that have multiple levels of formality expressed through honorifics or "grammatical register." for example, the sentence 'are you sure?' can have two possible correct translations in german: 'sind sie sicher?' for the formal register and 'bist du sicher?' for the informal one. leaving the model to choose between different valid translation options can lead to translations with inconsistent tone that are perceived as inappropriate by users depending on their demographics and cultural backgrounds, in particular for certain use cases (e.g. customer service, business, gaming chat). most prior research addressing this problem has been tailored to individual languages and proposed custom models trained on data with consistent formality @xcite , or through side constraints to control politeness or formality @xcite @xcite . ## isometric slt isometric translation is the task of generating translations similar in length to the source input @xcite . as a new research area in machine translation, this is the first time isometric translation is proposed as a shared task. 47 we considered 3 translations directions (english -german, english-french and english-spanish) and 2 training conditions: constrained and unconstrained.