Unsupervised Subtitle Segmentation with Masked Language Models (2023)

Abstract: We describe a novel unsupervised approach to subtitle segmentation, based on pretrained masked language models, where line endings and subtitle breaks are predicted according to the likelihood of punctuation to occur at candidate segmentation points. Our approach obtained competitive results in terms of segmentation accuracy across metrics, while also fully preserving the original text and complying with length constraints. Although supervised models trained on in-domain data and with access to source audio information can provide better segmentation accuracy, our approach is highly portable across languages and domains and may constitute a robust off-the-shelf solution for subtitle segmentation.

## introduction subtitling is one of the principal means of providing accessible audiovisual content. with the ever increasing production of audiovisual content in multiple domains and languages, in the current digital era, subtitle provision can benefit from automation support, via automatic speech recognition and/or machine translation @xcite @xcite @xcite . subtitles are subject to specific constraints in order to achieve adequate readability, including layout, on-screen duration and text editing. among these constraints, segmentation addresses the maximum number of characters per line, the number of lines per subtitle, and breaks at natural linguistic frontiers. segmentation has been shown to be an important readability factor @xcite , with improperly segmented subtitles resulting in increased cognitive effort and reading times for users. thus, automated subtitling systems need to generate properly segmented subtitles to achieve readability. * these authors contributed equally to this work. a typical baseline for subtitle segmentation, still used in some production systems, is simple character counting, whereby line breaks are inserted before reaching the maximum allowed number of characters per line. although simple and fast, this approach does not address the need for linguistically correct segments and, therefore, falls short in terms of readability. several approaches have been proposed to improve segmentation by automated means. @xcite proposed a machine learning method where subtitle breaks are predicted by support vector machine and linear regression models trained on professionally-created subtitles. a similar method based on conditional random fields was then shown to improve over these results @xcite . approaches that directly generate subtitle breaks within neural machine translation have also been proposed in recent years @xcite . recently, @xcite developed a multilingual segmenter which generates both text and breaks and may be trained on textual input only, or on joint text and audio data. although quality subtitle segmentation may be achieved with the aforementioned approaches, they require supervised training on segmented subtitle corpora. at present, the largest subtitle corpus is open subtitles @xcite , which mainly covers entertainment material, contains subtitles mostly created by non-professionals or automatically translated, and does not include line breaks. the must-cinema corpus @xcite , on the other hand, is a multilingual speech translation corpus that includes subtitles breaks, but is only available for 8 languages at the moment. considering the vast amount of languages and domains in audiovisual content, the lack of segmented training data hinders the development of robust automated subtitling systems. in this work, we describe a novel unsupervised method based on pretrained masked language mod-els (mlm), where line and subtitle breaks are inserted according to the likelihood of a segment acting as an isolated unit, as approximated by the probability of a punctuation mark occurring at a given segmentation point. in our experiments, this novel approach obtained competitive results on most metrics, while also fully preserving the original text and complying with length constraints. our system may thus be used as a simple yet efficient subtitle segmenter with any pretrained masked language model, for any language covered by the model. ## approach our approach is based on the standard view that the more appropriate subtitle segments are those that may function as isolated grammatical chunks. we further hypothesise that a relevant approximation for the identification of this type of unit is the likelihood of a punctuation mark being inserted at the end of a candidate segment, as punctuation may mark the closure of a syntactic unit and is often associated with discursive pauses. to test this hypothesis, we compute the likelihood of punctuation marks at different segmentation points, as predicted by a pretrained mlm, and select the insertion point with the highest likelihood. 1 the segmentation candidates are determined under a sliding-window approach over the entire input text. we first generate the list of all @xmath0, β> over the unprocessed portion of the text, where α is a segmentation candidate of length under a specified limit k, corresponding to the maximum number of characters per line, and β is the remaining portion of the text to be segmented. we then score all segmentation candidates α with one of the lm scoring variants described below. a segmentation marker, either end-of-line (<@xmath1, or end-of-block indicating the end of a subtitle (<@xmath2, is then appended to the best scoring candidate, and β becomes the input text to be segmented in a recursive iteration of the process. since our method does not rely on any additional information, such as an audio source, to determine the segmentation type, @xmath3; otherwise, @xmath4. we thus generate subtitles with a maximum of two lines, following a standard recommendation in subtitling. we also define a minimal number of characters (min) in α for the segmentation process to apply, and do not segment lines that are under the specified character limit. we evaluated three approaches to compute segmentation scores over each candidate @xmath5, β>: the first two methods are variants of our core approach. the third method, while also based on the same pretrained mlm, relies instead on the pseudoperplexity of the sequences according to the mlm, computed following @xcite . we included this latter variant to measure the potential of using lm scoring directly, without resorting to the likelihood of punctuation marks. ## experimental setup corpora. for all experiments, we used the mustst-cinema corpus @xcite , which is derived from ted talks and contains both line and subtitle break markers. in addition to being publicly available, it also allows for a direct comparison with the supervised models of @xcite . we report results of our approach on the 6 must-cinema datasets for which comparative results were available, directly predicting segmentation on the test sets without any training. 2 methods. for our approach, we tested the three variants described in section 2. we used bert @xcite as our mlm for all languages. 3 . additionally, we included a variant called overt clueing (oc), where an overt punctuation mark at the end of a candidate segment increments the mask score by 1. we then compared the results of the best lm-based variant with those obtained by alternative approaches. in all cases, our results were computed with @xmath6, as this value obtained the best results overall over the development sets, although the differences were minor with the other values we tested (1, 10 and 20). 4 we used the simple character counting approach (hereafter, countchars) as baseline, and, as representative supervised methods on the selected datasets, the models described by @xcite . their core supervised approach is based on a transformer @xcite architecture with 3 encoder layers and 3 decoder layers, trained on textual must-cinema input only (mc.text), or on complementary audio data as well via an additional speech encoder with 12 encoder layers (mc.multi). they trained each variant on either monolingual data alone (mono), or in a multilingual setting (multi). finally, they also report results for a variant (os.text) trained on the open subtitles corpus @xcite for their zero-shot experiments. evaluation. we use the subtitle-oriented metric sigma @xcite , which computes the ratio of achieved bleu @xcite over an approximated upper-bound bleu score, on text that includes line and subtitle breaks. sigma is meant to support the evaluation of imperfect texts, i.e. text that differs from the reference when breaks are omitted. although our approach does not produce imperfect text, achieving perfect bleu scores when breaks are ignored, we used this metric for comparison purposes. we also report break coverage results @xcite , defined as the ratio of predicted breaks over reference breaks, which we computed separately for the eol and eob breaks. finally, we include length conformity results (cpl), measured as the percentage of subtitle lines whose length is under the maximum number of characters defined by the subtitle guidelines (42 in the ted guidelines foot_4 ). ## lm-based segmentation variants we first compared the three methods described in section 2 on the english, spanish and german datasets, with the results described in ## comparative results in in overall, overt clueing improved over our core method by an average of 3.12 sigma points, indicating that some likely punctuation configurations were not properly captured by our mlm approximation. in general, our approach tends to overgenerate eol markers, whereas the opposite is true for the selected supervised models. determining which of these tendencies leads to better subtitle readability would require a specific human evaluation which we leave for future research. although the zero-shot sigma results obtained by the supervised mc.multi method show the potential of this approach to provide pretrained models applicable to other languages, two important aspects are worth considering. first, the available zero-shot results were obtained on datasets in the same domain as the data seen to train the supervised models. a more complete assessment of the capabilities of these models in zero-shot settings, which would be the most frequent scenario consid-ering the lack of training data across domains and languages, would require specific evaluations in other domains. secondly, although segmentation is a key aspect for subtitle readability, length conformity is an equally important constraint, if not more so considering that subtitles with lines over the cpl limit are considered invalid in subtitling. our proposed unsupervised method can thus be seen as a pragmatic approach which guarantees valid subtitles while also providing quality segmentation across the board. foot_6 ## conclusions we described an unsupervised approach to subtitle segmentation, based on pretrained masked language models, where line or subtitle breaks are inserted according to the likelihood of punctuation occurring at candidate segmentation points. although supervised models, trained on indomain data with audio support, were shown to perform better that this simple textual approach in terms of the sigma metric, they tend to generate imperfect text to varying degrees, while also failing to fully meet length constraints that are essential for subtitling. in contrast, our lm-based textual approach outperformed supervised models in most cases in terms of break generation coverage, while also fully preserving the original text, complying with length constraints, and obtaining competitive results in terms of sigma. this simple approach may thus provide a highly portable complementary solution for subtitle segmentation across languages and domains.