Towards a methodology for evaluating automatic subtitling (2022)

Abstract: In response to the increasing interest towards automatic subtitling, this EAMT-funded project aimed at collecting subtitle post-editing data in a real use case scenario where professional subtitlers edit automatically generated subtitles. The post-editing setting includes, for the first time, automatic generation of timestamps and segmentation, and focuses on the effect of timing and segmentation edits on the post-editing process. The collected data will serve as the basis for investigating how subtitlers interact with automatic subtitling and for devising evaluation methods geared to the multimodal nature and formal requirements of subtitling.

## project overview automatic subtitling is the task of generating target language subtitles for a given video without any intermediate human transcription and timing of the source speech. the source speech in the video is automatically transcribed, translated and segmented into subtitles, which are synchronised with the speech -a process called automatic spotting (or auto-spotting). automatic subtitling is becoming a task of increasing interest for the mt community, practitioners and the audiovisual industry. despite the technological advancements, the evaluation of automatic subtitling still represents a significant research gap. popular mt evaluation metrics consider only content-related parameters (translation quality), but not form-related © 2022 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. parameters, such as format (length and segmentation) and timing (synchronisation with speech, reading speed), which are important features for high-quality subtitles @xcite . moreover, the way subtitlers interact with automatically generated subtitles has not been yet explored, since the majority of works which conducted human evaluations of the post-editing effort in mt for subtitling have focused on edits in the textual content @xcite @xcite . this project seeks to investigate automatic subtitling, the factors contributing to post-editing effort and their relation to the quality of the output. this is achieved through the collection of rich, product-and process-based subtitling data in a real use case scenario where professional subtitlers edit automatically translated, spotted and segmented subtitles in a dedicated subtitling environment. the richness of the data collected during this one-year project is ideal for understanding the operations performed by subtitlers while they interact with automatic subtitling in their professional environment and for applying mixed methods approaches to: for each subtitler, we collected the following data: 1) original automatically-generated subtitle files and the corresponding final human postedited subtitle files in subrip .srt format; 2) process logs from the matesub tool, which records the original and final subtitle, original and final timestamps and total time spent on the subtitle; 3) keystrokes, using inputlog 3 @xcite . screen recordings were also collected to trace the translation and segmentation decisions of the subtitlers and identify possible outliers. at the end of the task, the subtitlers completed a questionnaire giving feedback on their user experience with automatic subtitling, particular problems faced, and their general impressions on automatic subtitling. for @xmath0, we collected in total 1,199 subtitles from the first subtitler (it1) and 1,208 subtitles from the second subtitler (it2), while for @xmath1,198 subtitles. based on the process logs we can define the status of each subtitle: new -a new subtitle is added by the subtitler; deleted -an automatically generated subtitle is discarded by the subtitler; or edited -any subtitle that is not new or deleted, regardless of whether it was confirmed exactly as generated by the system or changed. ## final remarks this project focuses on automatic subtitling and the challenges in its evaluation due to the multi-1 https://ict.fbk.eu/must-cinema/ 2 https://matesub.com/ 3 https://www.inputlog.net/ modal nature of the source medium (video, audio) and the formal requirements of the target (format and timing of subtitles). the data collected constitute the basis for future multi-faceted analyses to explore correlations between translation quality, spotting quality, and post-editing effort, possibly leading to new metrics for automatic subtitling. the subtitling data collected will be publicly released to promote research in automatic subtitling.