SBAAM ! Eliminating Transcript Dependency in Automatic Subtitling (2024)

Abstract: Subtitling plays a crucial role in enhancing the accessibility of audiovisual content and encompasses three primary subtasks: translating spoken dialogue, segmenting translations into concise textual units, and estimating timestamps that govern their on-screen duration. Past attempts to automate this process rely, to varying degrees, on automatic transcripts, employed diversely for the three subtasks. In response to the acknowledged limitations associated with this reliance on transcripts, recent research has shifted towards transcription-free solutions for translation and segmentation, leaving the direct generation of timestamps as uncharted territory. To fill this gap, we introduce the first direct model capable of producing automatic subtitles, entirely eliminating any dependence on intermediate transcripts also for timestamp prediction. Experimental results, backed by manual evaluation, showcase our solution’s new state-of-the-art performance across multiple language pairs and diverse conditions.

## introduction subtitling aims to facilitate the accessibility of audiovisual media, such as movies, tv shows, and video lectures, by providing users with a textual translation of spoken content. subtitles consist of two components: a textual block, typically encompassing one or two lines, and its corresponding time duration, indicated by start and end timestamps. to ensure effective on-screen presentation and minimize users' cognitive load, subtitles should conform to spatio-temporal constraints @xcite . these include restrictions on the maximum number of characters per line and a display duration that guarantees synchronization with the video while granting viewers sufficient time to read the entire text. automating the task involves addressing three main subtasks: translation of the spoken content, segmentation of the translated text into blocks and lines, and estimation of the timestamps for each block. early approaches to automatic subtitling (as) adopted a cascade architecture @xcite @xcite , i.e. a pipeline of components, including automatic speech recognition (asr) and machine translation (mt) models. in these systems, transcripts served as the foundational element for all three subtasks, despite the well-documented limitations of this reliance on them, such as error propagation @xcite , the loss of useful prosody information @xcite , inapplicability to source languages lacking written forms @xcite , and higher computational and environmental cost @xcite due to the need to run multiple models. to cope with these limitations, subsequent studies aimed to streamline the subtitling pipeline by reducing its dependency on transcripts. in this endeavor, direct speech-to-text translation (st) systems @xcite , capable of translating speech without recourse to intermediate symbolic representations, were successfully used by @xcite for the translation step (subtask ). for subtitle segmentation (subtask ), efforts focused on adapting mt @xcite @xcite @xcite and language models @xcite to directly produce translations that incorporate block and line boundaries as specific tags, also by exploiting audio information @xcite . to date, compared to subtasks and , timestamp estimation (subtask ) has received much less attention. this regards not only the elimination of intermediate transcription steps, the primary focus of this work, but also evaluation, as there is still no reliable and informative metric for directly assessing the quality of the predicted timestamps. as for the task itself, current approaches, including those exclusively based on direct st models @xcite , still require transcripts for timestamp estimation, which involves i) generating captions (i.e., segmented transcripts), ii) estimating timestamps for caption blocks, typically through a connectionist temporal classification (ctc) loss @xcite , and iii) projecting them onto target subtitles. regarding automatic evaluation, current metrics @xcite are by design holistic and therefore inadequate to precisely measure timestamp estimation quality. to bridge these gaps, we introduce and evaluate the first fully end-to-end as solution foot_0 that seamlessly produces both subtitles (i.e., segmented translations) and their timestamps without any reliance on intermediate transcripts. this innovation is realized by incorporating into our model the capability to directly determine the temporal alignment between the spoken utterances and their corresponding translation in subtitle form. along this direction, our contributions are the following: ## direct model for automatic subtitling following previous work on direct st @xcite @xcite , we build a direct autoregressive encoder-decoder model, where the encoder is composed of three blocks: i) an acoustic encoder -made of two 1d convolutions with stride 2 and n a conformer (gulati et al., 2020) layers, ii) a length adaptor -a ctc compression @xcite module that averages the vectors corresponding to the same ctc prediction, and iii) a semantic encoder -made of n s conformer layers. the encoder output is then fed to an autoregressive decoder d and, in parallel, to a ctc on target (tgtctc) module @xcite . the full architecture is shown in figure 1 . we train our model with a composite loss (l), which is a linear combination of a label smoothing cross-entropy (ce) loss @xcite on the decoder d, a ctc loss on the tgtctc module, and a ctc loss on the ctc compression module: at inference time, subtitles are generated by estimating their likelihood with the joint ctc/attention framework with ctc rescoring @xcite through a linear combination of the probabilities of the tgtctc module and the decoder: ## subsonar since it was proposed, the suber foot_4 metric @xcite has been used to provide a holistic evaluation of subtitles, encompassing translation quality, block/line segmentation accuracy, and timing. specifically, suber computes the number of word edits and block/line edits required to match the reference, where hypothesis and reference words are allowed to match only within subtitle blocks that overlap in time. this definition highlights how suber is sensitive only to major errors in terms of timing, as it solely checks whether two blocks overlap, even if only for a few milliseconds. this limitation motivates our proposal of subsonar, a new metric designed to be more sensitive to time shifts and, consequently, more suitable for evaluating the quality of subtitle timestamps. to this aim, we leverage the multimodal and multilingual sonar model @xcite , designed to generate sentence embeddings within a shared multimodal (text and audio) semantic space for all languages. specifically, we calculate the cosine similarity between the sonar embeddings of the text within a subtitle block and its corresponding audio, determined by the timestamp of the block. subsequently, we average the similarity scores across all subtitle blocks, which results in a single score in the [-1, 1] range. being sonar trained to capture both text and audio semantics, higher subsonar scores indicate better alignment between text and audio content, which is influenced by both translation quality and timestamp accuracy. nevertheless, as revealed by the empirical validation presented in the following sections ( §5,6), sub-sonar exhibits higher sensitivity to timing accuracy than to translation quality. ## automatic evaluation in this section, we evaluate our solutions automatically through comparative experiments in different resource conditions and language settings. first, we validate the adoption of the joint ctc rescoring and compare the timestamp estimation methods introduced in §3 using a multilingual system trained and tested on all the 7 language pairs (@xmath0,es,fr,it,nl,pt,ro}) of must-cinema @xcite ) ( §5.1). then, we confirm their strength by comparing two bilingual systems (en-de and en-es) trained in the high resource conditions of the iwslt 2023 subtitling track @xcite with the results reported for the current state of the art in direct as and production tools on several test sets ( §5.2). lastly, we demonstrate that our solutions close the gap with the cascade approach, outperforming the best iwslt cascade models on the 4 publicly available validation sets ( §5.3). full experimental settings and details about training data are given in appendix a to ensure the reproducibility of our work. ## manual evaluation to corroborate the findings from our automatic evaluation, we conducted the first-ever manual evaluation of timestamp quality in subtitling. this evaluation was specifically conducted on the two language pairs addressed in our high-resource experiments (en-de, en-es). to focus only on timestamp quality, we compared different timestamp estimation techniques on the same automatic translations. specifically, we selected the outputs of our high-resource systems with joint ctc rescoring and confronted the baseline lev method and our proposed sbaam. for each language pair, the manual evaluation set -on which the two systems were run -is composed of subsets of the ec short clips and must-cinema test sets, for a total of 22 videos corresponding to approximately 1 hour of audio (see appendix c.1 for details on the selected videos). the evaluation was carried out by two annotators per language pair, who are proficient in english (c1) as well as native speakers or very proficient (c2) in the target language (german or spanish). 10 we instructed the annotators with ad-hoc guidelines (described in appendix c.2) to adjust the start, end, or both timestamps when the generated subtitles are not synchronized with the speech. to collect this information, we used elan @xcite , an audio/video annotation tool commonly employed in the literature @xcite . to cope with the inherent assessors' subjectivity (i.e. being more aggressive/tolerant in deciding whether a timing is acceptable or has to be edited) the outputs of the two timestamp estimation methods were randomly assigned to the two annotators, ensuring that each one worked on all 22 audios and annotated half of the outputs from both methods. to prove the quality of the manual evaluation and better understand the difficulty of the task, for each language pair, 25% of the test set was double-annotated. we calculated cohen's kappa @xcite to measure the inter-annotator agreement on whether a start/end timestamp has to be edited or not. the resulting values -0.65 for en-de and 0.61 for en-es -indicate a "substantial" agreement (landis and @xcite , confirming both en-de model time shift (ms) % edited ts start end avg lev 544 ± 718 38.47 40.74 39.61 sbaam 321 ± 453 18.43 18.81 18.62 en-es model time shift (ms) % edited ts start end avg lev 520 ± 626 38.01 39.45 38.73 sbaam 347 ± 570 11.17 12.89 12.03 we can conclude that our manual analysis not only confirms the results obtained through the automatic subsonar metric, but also further highlights the efficacy of our proposed timestamp estimation method and its substantial superiority over previous solutions. ## conclusions in recent years, research on automatic subtitling has shifted toward direct models that do not rely on the transcription of the input audio. the potential advantages of a transcription-free approach have motivated the use of direct models for the translation and segmentation steps, leaving unexplored the direct generation of timestamps on the target text. in this paper, we filled this gap by introducing the first direct system that does not require the generation of transcripts/captions in any phase of the process, including subtitle timestamp estimation. with the introduction of a new metric, subsonar, dedicated to evaluating timestamp quality, and experiments on different domains and 7 language pairs, we demonstrated the effectiveness of our solution, which was further validated by a dedicated manual evaluation. lastly, we showed that our model closed the gap with the cascade paradigm, approaching and even outperforming the best cascade architectures from the last iwslt campaign, thereby setting a new state of the art for most of the publicly available benchmarks.