FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN (2020)

Abstract: The evaluation campaign of the International Conference on Spoken Language Translation (IWSLT 2020) featured this year six challenge tracks: (i) Simultaneous speech translation, (ii) Video speech translation, (iii) Offline speech translation, (iv) Conversational speech translation, (v) Open domain translation, and (vi) Non-native speech translation. A total of teams participated in at least one of the tracks. This paper introduces each track’s goal, data and evaluation metrics, and reports the results of the received submissions.

## introduction the international conference on spoken language translation (iwslt) is an annual scientific conference @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite for the study, development and evaluation of spoken language translation technology, including: speechto-text, speech-to-speech translation, simultaneous and consecutive translation, speech dubbing, cross-lingual communication including all multi-modal, emotional, para-linguistic, and stylistic aspects and their applications in the field. the goal of the conference is to organize evaluations and sessions around challenge areas, and to present scientific work and system descriptions. this paper reports on the evaluation campaign organized by iwslt 2020, which features six challenge tracks: the challenge tracks were attended by 30 participants (see ## simultaneous speech translation simultaneous machine translation has become an increasingly popular topic in recent years. in particular, simultaneous speech translation enables interesting applications such as subtitle translations for a live event or real-time video-call translations. the goal of this challenge is to examine systems for translating text or audio in a source language into text in a target language from the perspective of both translation quality and latency. ## video speech translation we are living the multiple modalities world in which we see objects, hear sounds, feel texture, smell odors, and so on. the purpose of this shared task is to ignite possibilities of multimodal machine translation. this shared task examines methods for combining video and audio sources as input of translation models. ## offline speech translation in continuity with last year @xcite , the offline speech translation task required participants to translate english audio data extracted from ted talks foot_4 into german. participants could submit translations produced by either cascade architectures (built on a pipeline of asr and mt components) or end-to-end models (neural solutions for the direct translation of the input audio), and were asked to specify, at submission time, which of the two architectural choices was made for their system. similar to last year, valid end-to-end submissions had to be obtained by models that: ## conversational speech translation in conversational speech, there are many phenomena which aren't present in well-formed text, such as disfluencies. disfluencies comprise e.g., filler words, repetitions, corrections, hesitations, or incomplete sentences. this differs strongly from typical machine translation training data. this mismatch needs to be accounted for when translating conversational speech both for domain mismatch as well as generating well-formed, fluent translations. while previously handled with intermediate processing steps, with the rise of end-toend models, how and when to incorporate such a pre-or post-processing steps between speech processing and machine translation is an open question. disfluency removal typically requires tokenlevel annotations for that language. however, most languages and translation corpora do not such annotations. using recently collected fluent references @xcite for the common fisher spanish-english dataset, this task poses several potential questions: how should disfluency removal be incorporated into current conversational speech translation models where translation may not be done in a pipeline, and can this be accomplished without training on explicit annotations? ## open domain translation the goals of this task were to further promote research on translation between asian languages, the exploitation of noisy parallel web corpora for mt, and thoughtful handling of data provenance. ## non-native speech translation the non-native speech translation task has been added to iwslt this year. the task focuses on the very frequent setting of non-native spontaneous speech in somewhat noisy conditions, one of the test files even contained speech transmitted through a remote conferencing platform. we were interested in submissions of both types: the standard two-stage pipeline (asr+mt, denoted "cascaded") as well as end-to-end ("e2e") solutions. this first year, we had english as the only source language and czech and german as the target languages. participants were allowed to submit just one of the target languages. the training data sets permitted for "constrained" submissions were agreed upon the training data with the offline translation task (section 4) so that task participants could reuse their systems in both tasks. participants were however also allowed to use any other training data, rendering their submissions "unconstrained".