Direct Models for Simultaneous Translation and Automatic Subtitling: FBK @ IWSLT 2023 (2023)

Abstract: This paper describes the FBK’s participation in the Simultaneous Translation and Automatic Subtitling tracks of the IWSLT 2023 Evaluation Campaign. Our submission focused on the use of direct architectures to perform both tasks: for the simultaneous one, we leveraged the knowledge already acquired by offline-trained models and directly applied a policy to obtain the real-time inference; for the subtitling one, we adapted the direct ST model to produce well-formed subtitles and exploited the same architecture to produce timestamps needed for the subtitle synchronization with audiovisual content. Our English-German SimulST system shows a reduced computational-aware latency compared to the one achieved by the top-ranked systems in the 2021 and 2022 rounds of the task, with gains of up to 3.5 BLEU. Our automatic subtitling system outperforms the only-existing solution based on a direct system by 3.7 and 1.7 SubER in English-German and English-Spanish respectively.

## introduction in recent years, the advances in natural language processing and machine learning led to a surge of interest in developing speech translation (st) systems that can translate speech from one language into text in another language without human intervention. significant progress has been specially made toward end-to-end st models @xcite trained to directly translate speech without the intermediate steps of transcription (through automatic speech recognition -asr) and translation (through machine translation -mt). along with this growing interest in direct st, also accompanied by a reduction of the performance gap with respect to cascaded architectures @xcite , other trends have emerged thanks to deep learning advancements, which made it possible to deploy direct solutions to perform the task in real-time (i.e. to produce partial translations while continuing to process the input audio) or to automatically generate subtitles for audiovisual content (i.e. pieces of translated text which have to conform to specific spatiotemporal constraints and be synchronized with the video). the international workshop on spoken language translation (iwslt) is playing an important role in advancing the state-of-the-art in these fields by organizing a series of evaluation campaigns @xcite @xcite focused on simultaneous speech translation (simulst) and, this year for the first time, automatic subtitling. these campaigns provide a unique opportunity for researchers to compare their systems against others, share their findings, and identify areas for further improvement. in this paper, we describe fbk's participation in the iwslt 2023 evaluation campaigns @xcite for simultaneous translation and automatic subtitling. motivated by the promising results reported in previous works @xcite , our approach is characterized by the use of direct st models to address both tasks. for the simultaneous speech-to-text translation (simulst) task, we participated in the @xmath0, as this has recently been shown not to be necessary to achieve competitive results @xcite . for the automatic subtitling task, we participated in both the @xmath1, as in @xcite . our results demonstrate the effectiveness of our approach. in simulst, the computational-aware latency of our models is lower compared to the winning systems of the last two rounds (2021, and 2022) of the iwslt simulst evaluation campaign, with gains up to 3.5 bleu. in automatic subtitling, our systems improve the results reported in @xcite which, to the best of our knowledge, represents the only-existing solution based on a direct model. specifically, on average among the various dev sets available for the task, we achieve 3.7 suber on en-de and 1.7 suber on en-es. ## applied direct models for this year's submission, we applied the direct st models to the two different scenarios of simultaneous translation and automatic subtitling. ## conclusions we presented the fbk's systems built to participate in the iwslt 2023 evaluation campaigns for simultaneous speech translation (en-de) and automatic subtitling (en-{de, es}). our submissions are characterized by the use of direct speech translation models to address both tasks, without any further modification nor adaptation for the simultaneous task, and with a fine-tuning on subtitle-like translations for the automatic subtitling task. our simulst system achieves a lower computationalaware latency with up to 3.5 bleu gain compared to the last two years' winners. our automatic subtitling system achieves 3.7 and 1.7 suber improvement on en-de and en-es respectively, compared to the only solution published in the literature based on a direct system.