FINDINGS OF THE IWSLT 2024 EVALUATION CAMPAIGN (2024)

Abstract: This paper reports on the shared tasks organized by the 21st IWSLT Conference. The shared tasks address 7 scientific challenges in spoken language translation: simultaneous and offline translation, automatic subtitling and dubbing, speech-to-speech translation, dialect and low-resource speech translation, and Indic languages. The shared tasks attracted 17 teams whose submissions are documented in 27 system papers. The growing interest towards spoken language translation is also witnessed by the constantly increasing number of shared task organizers and contributors to the overview paper, almost evenly distributed across industry and academia.

## introduction the international conference on spoken language translation (iwslt) is the premier annual scientific conference for all aspects of spoken language translation (slt). iwslt is organized by the special interest group on spoken language translation (sigslt), which is supported by acl, isca and elra. like in all the previous 20 editions, this year's conference was preceded by an evaluation campaign featuring shared tasks addressing scientific challenges in slt. this paper reports on the 2024 iwslt evaluation campaign, which offered the following 7 shared tasks: the shared tasks attracted 18 teams (see ## simultaneous slt simultaneous speech translation focuses on translating speech in real-time, in manner vaguely similar to simultaneous interpreting. the system is designed to begin translating before the speaker has finished their sentence. this technology is particularly useful in scenarios such as international conferences, personal travel, or public emergency events. this year, the task included two tracks: speechto-text and speech-to-speech, covering four language directions: english to german, english to chinese, english to japanese, and czech to english-a new language direction added this year. ## automatic subtitling in recent years, the task of automatically creating subtitles for audiovisual content in another language has gained a lot of attention due to the rapid increase in the global distribution and streaming of movies, series, and user-generated videos. reflecting these trends, the automatic subtitling track was introduced for the first time in 2023 as part of the iwslt evaluation campaigns. given the growing interest in this area, the task has been continued this year with the addition of a new subtrack, subtitle compression, alongside the existing automatic subtitling sub-task from the previous edition. in the automatic subtitling task, participants were asked to generate subtitles in german and/or spanish from english speech in audiovisual docu-ments. in the new subtitle compression task, participants were required to automatically rephrase subtitles that did not comply with the reading speed constraint (i.e., subtitles exceeding a certain length/time ratio given in characters per second) to ensure they met the required standards. the decision to have works focusing on this specific aspect of subtitling is highly motivated by the existing requirements posed by subtitles providers @xcite . in fact, the constraint on the reading speed is a commonly adopted standard to ensure that viewers can enjoy audiovisual content without experiencing fatigue or distraction due to excessive reading demands @xcite . therefore, adhering to this limit is crucial, making the development of ad-hoc methods to improve automatically generated subtitles that exceed this threshold of particular interest. ## low-resource slt the 4 th edition of the low-resource spoken language translation track focused on the translation of speech from a variety of data-scarce languages. the target language is typically a higher-resource one, generally of similar geographical or historical linkages. the goal of this shared task is to benchmark and promote speech translation technology for a diverse range of dialects and low-resource languages. while significant research progress has been demonstrated recently, many of the world's languages and dialects lack the parallel data at scale needed for standard supervised learning. ## indic languages track in the realm of spoken language processing, speech-to-text translation (st) holds a crucial role at the intersection of natural language processing. the primary aim of st is to convert spoken language from one linguistic context into written text in another language. this typically involves using automatic speech recognition (asr) to convert speech in the source language into text, followed by machine translation (mt) to translate the source language text into the target language. st is a multimodal task that takes speech input and produces output in text format. furthermore, it is inherently multilingual, taking speech input in one language and generating text output in another. traditionally, human language translators proficient in both the source and target languages have handled this task. however, the scarcity of translators fluent in multiple languages has created a pressing need for a dedicated model tailored to excel in the unique realm of st tasks across diverse languages. recent advancements in st have predominantly focused on high-resource languages, leaving a significant gap for low-resource languages that face a substantial catch-up journey. the attention imbalance is primarily due to the scarcity of data for low-resource languages, as most deep-learning models depend on data abundance. acquiring such data for low-resource languages poses a formidable challenge. while a considerable body of research is dedicated to st across diverse language families, a noticeable gap exists in investigating this domain concerning low-resource indian languages. currently, there are no datasets specifically designed for the st task in indian languages, covering both the indo-aryan and the dravidian language families. this research aims to create either an end-to-end (e2e) or a cascaded st model this indic track aims to establish an st translation model that spans a diverse array of dialects and low-resource languages originating from the indo-aryan and dravidian language families in india. given that a significant portion of the data is sourced from very low-resource languages, these languages remain largely unexplored in the realm of speech translation. compounding this challenge is the fact that many of the target languages are distantly related to english. consequently, we anticipate that relying solely on pre-trained speech is partially matching speakers lips and/or is a bit natural as well as the meaning of the source sentence are adequately transferred into the target language. 4 speech naturalness is of acceptable quality and the meaning of the source sentence is mostly preserved. 5 speech is mostly natural and the translation is almost perfect or is a good paraphrase of reference. 6 speech looks completely natural and the translation is perfect in every sense of the word. models may encounter numerous obstacles. the dataset provided will serve as the inaugural benchmark and gold standard dataset, encompassing all indian languages. we aspire for participants to develop systems capable of real-world deployment in the future.