Direct Speech Translation for Automatic Subtitling (2023)

Abstract: Automatic subtitling is the task of automatically translating the speech of audiovisual content into short pieces of timed text, i.e., subtitles and their corresponding timestamps. The generated subtitles need to conform to space and time requirements, while being synchronized with the speech and segmented in a way that facilitates comprehension. Given its considerable complexity, the task has so far been addressed through a pipeline of components that separately deal with transcribing, translating, and segmenting text into subtitles, as well as predicting timestamps. In this paper, we propose the first direct speech translation model for automatic subtitling that generates subtitles in the target language along with their timestamps with a single model. Our experiments on 7 language pairs show that our approach outperforms a cascade system in the same data condition, also being competitive with production tools on both in-domain and newly released out-domain benchmarks covering new scenarios.

## introduction with the growth of websites and streaming platforms such as youtube and netflix, 1 the amount of audiovisual content available online has dramatically increased. suffice to say that the number of hours of netflix original content has increased by 2,400% from 2014 to 2019. 2 this phenomenon has led to a huge demand for subtitles, which is becoming more and more difficult to satisfy only with human resources. consequently, automatic subtitling tools are spreading to reduce subtitlers' workload by providing them with suggested subtitles to be post-edited @xcite . in general, subtitles can be either intralingual (hereinafter captions), if source audio and subtitle text are in the same language, or interlingual (hereinafter subtitles), if the text is in a different language. in this paper, we focus on automatizing interlingual subtitling, framing it as a speech translation (st) for subtitling problem. differently from st, in automatic subtitling the generated text has to comply with multiple requirements related to its length, format, and the time it should be displayed on the screen @xcite . these requirements, which depend on the type of video content and target language, are dictated by the need to keep users' cognitive effort as low as possible while maximizing comprehension and engagement @xcite . this often leads to a condensation of the original spoken content, aimed at reducing the time required for reading subtitles while increasing that of watching the video @xcite . being such a complex task, automatic subtitling has so far been addressed by dividing the process into different steps @xcite @xcite @xcite : automatic speech recognition (asr), timestamp extraction from audio, segmentation into captions, and their machine translation (mt) into the final subtitles. more recently, drawing from the evidence that direct models achieve competitive quality with cascade architectures @xcite , @xcite proposed an st system that jointly translates and segments into subtitles, arguing that direct models are able to better exploit speech cues and prosody in subtitle segmentation. however, their system does not generate timestamps, hence missing a critical aspect to reach the goal of fully automatic subtitling. furthermore, the current lack of benchmarks hinders a thorough evaluation of the technologies developed for automatic subtitling. in fact, the only corpus publicly available to date is must-cinema @xcite , which contains only single-speaker audio in the ted-talks domain with verbatim translations. to fill these gaps, this paper presents the first automatic subtitling system that performs the whole task with a single direct st model, and introduces two new benchmarks. our contributions can be summarized as follows: ## direct speech translation for subtitling motivated by all the advantages discussed in §2.1 and §2.3, we build the first automatic subtitling system solely based on a direct st model (figure 1 ). our system works as follows: i) the audio is fed to a subtitle generator ( §3.1) that produces the (untimed) subtitle blocks; ii) the computed encoder representations are passed to the source timestamp generator ( §3.2) to obtain the caption blocks and their corresponding timestamps; iii) the subtitle timestamps are estimated by the source-to-target timestamp projection ( §3.3) from the generated subtitles, captions, and source timestamps. these modules are described in the rest of this section. ## results in this section, we first ( §6.1) choose the best timestamp projection method among those introduced in §3.3. then ( §6.2), we compare the cascade and direct approaches trained in the same data conditions. lastly ( §6.3), we show that our 22 case:mixed|eff:no|tok:13a|smooth:exp|version:2.3.1. 23 we used version 1.1 of the script adopted for the iwslt subtitling task ( https://iwslt.org/2023 /subtitling ): https://github.com/hlt-mt/fbk -fairseq/blob/master/examples/speech to text /scripts/subtitle compliance.py . ## conclusions in this paper, we proposed the first approach based on direct speech-to-text translation models to fully automatize the subtitling process, including translation, segmentation into subtitles, and timestamp estimation. experiments in constrained data conditions on 7 language pairs demonstrated the potential of our approach, which outperformed the current cascade architectures with a ∼7% improvement in terms of suber. in addition, to test the generalizability of our findings across subtitling genres, we extended our evaluation setting by collecting two new test sets for @xmath0, es} covering different domains, degrees of subtitle condensation, and audio conditions. finally, we compared our models with production tools in unconstrained data conditions on both existing benchmarks and the newly collected test sets. this comparison further highlighted that our approach represents a promising direction: although trained on a relatively limited amount of data, our systems achieved comparable quality with production tools, with improvements in suber ranging from 0.2 to 5.0 on @xmath1.6 to 13.0 on @xmath2.