S ub ER - A Metric for Automatic Evaluation of Subtitle Quality (2022)

Abstract: This paper addresses the problem of evaluating the quality of automatically generated subtitles, which includes not only the quality of the machine-transcribed or translated speech, but also the quality of line segmentation and subtitle timing. We propose SubER - a single novel metric based on edit distance with shifts that takes all of these subtitle properties into account. We compare it to existing metrics for evaluating transcription, translation, and subtitle quality. A careful human evaluation in a post-editing scenario shows that the new metric has a high correlation with the post-editing effort and direct human assessment scores, outperforming baseline metrics considering only the subtitle text, such as WER and BLEU, and existing methods to integrate segmentation and timing features.

## introduction the use of automatically created subtitles has become popular due to improved speech recognition (asr) and machine translation (mt) quality in recent years. most notably, they are used on the web to make content available to a broad audience in a cost-efficient and scalable way. they also gain attraction in the media industry, where they can be an aid to professional subtitlers and lead to increased productivity. in this work, we address the problem of measuring the quality of such automatic subtitling systems. we argue that existing metrics which compare the plain text output of an asr or mt system to a reference text are not sufficient to reflect the particularities of the subtitling task. we consider two use cases: 1) running speech recognition on the audio track of a video to create subtitles in the original language; 2) translating existing subtitle files with an mt system. for the first case, the word error rate (wer) of the asr system is a natural choice for quality control. for mt there exist a wider range of automatic metrics such as bleu @xcite , ter @xcite , chrf @xcite and, more recently, learned metrics like bertscore @xcite and comet @xcite . these existing metrics are suited to measure the quality of asr and mt in terms of recognized or translated content only. however, subtitles are defined by more than just their textual content: they include timing information, as well as formatting with possible line breaks within a sentence in syntactically and semantically proper positions. figure 1 shows examples of subtitle files in the common subrip text (srt) format. evidently, it differs from plain text, in particular: all of these additional characteristics are crucial for the viewers' comprehension of the content. professional subtitlers check and possibly improve them as part of the machine-assisted process of subtitle creation. to assess the quality of automatically created subtitle files, it is beneficial to have a single metric that evaluates the asr/mt quality and the quality of the characteristics listed above. the main contributions of this work are: none of the above metrics, however, are automatic ones. they require manual evaluation by an expert to categorize errors and assign appropriate penalties depending on their severity. this makes their use costly and time-consuming. in this work we therefore address automatic quality assessment of subtitle files by comparing them to a professionally created reference. ## human evaluation to analyze the expressiveness of suber we conduct a human post-editing experiment on both subtitles automatically generated from audio, as well as automatic translations of subtitle text files. for each of the two post-editing tasks we employ three professional subtitlers with multiple years of experience in the subtitling industry. we evaluate how well automatic metric scores correlate with their post-editing effort and their mt quality judgements. there exists previous work measuring the productivity gains from post-editing automatic subtitles under the aspect of mt quality @xcite @xcite ) and segmentation quality @xcite @xcite , but to the best of our knowledge we conduct the first study with the goal of evaluating an automatic quality metric for subtitling. ## release of code we release the code to calculate the suber metric as part of an open-source subtitle evaluation toolkit foot_1 to encourage its use in the research community as well as the media industry and to further promote research of automatic subtitling systems. in addition to suber, the toolkit implements all baseline metrics used in all metrics can be calculated directly from srt input files. support for other subtitle file formats will be added on demand. ## conclusion in this work, we proposed suber -a novel metric for evaluating quality of automatically generated intralingual and interlingual subtitles. the metric is based on edit distance with shifts, but considers not only the automatically transcribed or translated text, but also subtitle timing and line segmentation information. it can be used to compare an automatically generated subtitle file to a human-generated one even if the two files contain a different number of subtitles with different timings. a thorough evaluation by professional subtitlers confirmed that suber correlates well with their transcription post-editing effort and direct assessment scores of translations. in most cases, suber shows highest correlation as compared to metrics that evaluate either the quality of the text alone, or use different approaches to integrate subtitle timing and segmentation information. the source code for suber will be publicly released for the benefit of speech recognition and speech translation research communities, as well as the media and entertainment industry.