Automatic Subtitling and Subtitle Compression: FBK at the IWSLT 2024 Subtitling track (2024)

Abstract: The paper describes the FBK submissions to the Subtitling track of the 2024 IWSLT Evaluation Campaign, which covers both the Automatic Subtitling and the Subtitle Compression task for two language pairs: English to German (en-de) and English to Spanish (en-es). For the Automatic Subtitling task, we submitted two systems: i) a direct model, trained in constrained conditions, that produces the SRT files from the audio without intermediate outputs (e.g., transcripts), and ii) a cascade solution that integrates only free-to-use components, either taken off-the-shelf or developed in-house. Results show that, on both language pairs, our direct model outperforms both cascade and direct systems trained in constrained conditions in last year’s edition of the campaign, while our cascade solution is competitive with the best 2023 runs. For the Subtitle Compression task, our primary submission involved prompting a Large Language Model (LLM) in zero-shot mode to shorten subtitles that exceed the reading speed limit of 21 characters per second. Our results highlight the challenges inherent in shrinking out-of-context sentence fragments that are automatically generated and potentially error-prone, underscoring the need for future studies to develop targeted solutions.

## introduction in response to the growing amount of audiovisual content produced every day, the task of automatically generating subtitles has seen increasing attention @xcite , with the goal of fostering the accessibility of the material by overcoming language barriers. in light of this, starting from the 2023 edition, the iwslt evaluation campaign includes the automatic subtitling task, in which participants had to generate well-formed subtitles in german and spanish starting from the corresponding english audio @xcite . in addition to requiring high-quality translations of the audio content, correct subtitles also need the translated text to be split into blocks (each of them possibly split into 2 lines) in a way that minimizes the users' cognitive effort @xcite @xcite , and these blocks have to be presented on-screen with the correct timing, i.e. in sync with the original audio. although there is no absolute rule to determine the cognitive effort required to read a subtitle, typical constraints to keep it low include: i) not having more than 2 lines per block (lpb); ii) keeping the number of characters per line (cpl) below a given threshold, which was set to 42 in the iwslt 2023 campaign; and iii) avoiding excessive reading speed expressed in the number of characters per second (cps) to be read by the user, which was set to 21. good subtitles should hence be displayed in text blocks that conform to these rules, and their adherence to the constraints can be measured as the percentage of blocks compliant with them. since automatic subtitling systems can fail in fully matching all the above constraints, the iwslt 2024 campaign introduced an additional subtitle compression sub-task, foot_0 which requires to reduce the number of characters in each block of pre-generated subtitles to an extent that satisfies the reading speed constraint, without compromising its semantic content. this paper describes fbk's submissions to both tasks (automatic subtitling and subtitle compression) of the iwslt 2024 subtitling track. our submitted systems cover both language directions under evaluation, namely english-german (en-de) and english-spanish (en-es). regarding automatic subtitling, we explored two approaches that led to two submissions, one for each training condition, constrained and unconstrained. on the one hand, following the promising results obtained by the first direct models for automatic subtitling @xcite , we trained a direct subtitling model ( §2.1) in constrained conditions, i.e. using only the data allowed by the organizers for this setting. we call this model direct as it generates the subtitles in the target languages (including block and line delimiters) as well as timestamps without any intermediate discrete content representation, such as textual transcripts of the audio. in this respect, it is different from the two direct models submitted in the 2023 edition as both required the generation of intermediate transcripts for the timestamps estimation, either by using an auxiliary automatic speech recognition (asr) system @xcite or by using auxiliary modules of the direct speech translation (st) system @xcite . on the other hand, we created a pipeline system ( §2.2) within the ai4culture foot_1 eu project, which binds us to use only code and models released under licenses as permissive as possible. lastly, our primary submission to the newly proposed subtitle compression task ( §2.3) tackled the problem with an llm-based approach. to this aim, we explored a first basic solution by prompting the model in zero-shot mode to shorten candidate hypotheses exceeding the 21 cps limit, and compared it with simpler, word/character deletion strategies. ## systems description in this section, we first describe the direct ( §2.1) and cascade ( §2.2) automatic subtitling systems, and then our subtitle compression submissions ( §2.3). ## results as a recap, fbk submitted the following runs: subtitling task • primary run in constrained condition: fbk drct 24 ( §2.1) • primary run in unconstrained condition: fbk cscd 24 ( §2.2) subtitle compression task • primary run: gpt ( §2.3, paragraph "user-oriented approach") • contrastive1 run: del all func wrds ( §2.3, "metric-oriented approach") • contrastive2 run: del art/prep/adv ( §2.3, "metric-oriented approach") ## conclusions we presented the fbk's submissions to the automatic subtitling and subtitle compression tasks of the iwslt 2024 evaluation campaign. for automatic subtitling, we proposed two systems: a direct model trained under constrained conditions and a cascade architecture integrating free-to-use components. our direct model showcased superior performance compared to constrained direct and cascade submissions of the last year. the cascade solution proved competitive with top-performing unconstrained and fine-tuned 2023 runs. for subtitle compression, our primary submission exploits gpt in zero-shot prompting mode to shorten subtitles exceeding the reading speed limit of 21 cps. while promising, this approach revealed the complexities of compressing out-of-context automatically generated sentence fragments, underscoring the necessity for further research in this area.