Dodging the Data Bottleneck: Automatic Subtitling with Automatically Segmented ST Corpora (2022)

Abstract: Speech translation for subtitling (SubST) is the task of automatically translating speech data into well-formed subtitles by inserting subtitle breaks compliant to specific displaying guidelines. Similar to speech translation (ST), model training requires parallel data comprising audio inputs paired with their textual translations. In SubST, however, the text has to be also annotated with subtitle breaks. So far, this requirement has represented a bottleneck for system development, as confirmed by the dearth of publicly available SubST corpora. To fill this gap, we propose a method to convert existing ST corpora into SubST resources without human intervention. We build a segmenter model that automatically segments texts into proper subtitles by exploiting audio and text in a multimodal fashion, achieving high segmentation quality in zero-shot conditions. Comparative experiments with SubST systems respectively trained on manual and automatic segmentations result in similar performance, showing the effectiveness of our approach.

## introduction massive amounts of audiovisual content are available online, and this abundance is accelerating with the spread of online communication during the covid-19 pandemic. the increased production of pre-recorded lectures, presentations, tutorials and other audiovisual products raises an unprecedented demand for subtitles in order to facilitate comprehension and inclusion of people without access to the source language speech. to keep up with such a demand, automatic solutions are seen as a useful support to the limited human workforce of trained professional subtitlers available worldwide @xcite . attempts to automatise subtitling have focused on machine translation for translating human-or automatically-generated source * work done when working at fbk. language subtitles @xcite @xcite . recently, direct st systems @xcite have been shown to achieve high performance while generating the translation in the target language without intermediate transcription steps. for automatic subtitling, @xcite suggested that, by directly generating target language subtitles from the audio (i.e. predicting subtitle breaks together with the translation), the model can improve subtitle segmentation by exploiting additional information like pauses and prosody. however, the scarcity of subst corpora makes it hard to build competitive systems for automatic subtitling, especially if no corpus is available for specific languages/domains. one solution to the subst data bottleneck could be leveraging st corpora by inserting subtitle breaks on their target side. automatic segmentation of a text into subtitles is normally implemented with rule-based approaches and heuristics, e.g. a break is inserted before a certain length limit is reached. more involved algorithms (svm, crf, seq2seq) predict breaks using a segmenter model trained on subtitling data for a particular language @xcite @xcite . still, the performance of these models relies on high-quality segmentation annotations for each language, which web-crawled subtitling corpora like opensubtitles @xcite rarely contain. in this work, we address the scarcity of subst corpora by developing a multimodal segmenter 1 able to automatically annotate existing st corpora with subtitle breaks in a zero-shot fashion. specifically, our segmenter exploits, for the first time in this scenario, the source language audio (here: en) and segmented target text already available in a few languages (here: de, en, fr, it). its key strength is the ability to segment not only target languages for which high-quality segmented data is available but also unseen languages having some degree of similarity with those covered by the original st resource(s). this opens up the possibility to automatically obtain synthetic subst training data for previously not available languages. along this direction, our zero-shot segmentation results on two unseen languages (es, nl) show that training a subst system on automatically-segmented data leads to comparable performance compared to using a gold, manually-segmented corpus. ## methodology our method to leverage st corpora for subst can be summarized as follows: i) we train different segmenters on available human-segmented subtitling data to select the best performing one; ii) we run the selected segmenter in a zero-shot fashion (i.e. without fine-tuning or adaptation) to insert subtitle breaks in unsegmented text data of unseen languages; iii) then, the automatically annotated texts are paired with the corresponding audio to obtain a synthetic parallel subst corpus; iv) finally, a subst model is trained on the synthetic corpus. we test our method on two language pairs (en-es, en-nl) by comparing the results of subst models trained on synthetic data with those of identical models trained on original gold data. ## zero-shot segmentation aiming to build a subst model for unseen languages (es, and nl), we first select the best segmenter for generating synthetic @xmath0. as shown in in comparison, the multimodal segmenter performs significantly better. it reaches an absolute gain of 6.1 sigma points for nl and 2.9 for es compared to textual. moreover, contrary to textual and supervised, the multimodal model learnt to perfectly copy the text, as shown by the high bleu scores (up to 99.9 on nl), close to the maximum score of a method -count chars -that by design does not change the original text. the cpl results are in agreement with bleu: for both languages, the multimodal model respects the length constraint in more than 91% of the subtitles. strikingly, even if the two target languages were never seen by the model, these results are similar to those obtained on seen languages (see ## subst with synthetic data since our multimodal segmenter achieves the best performance overall, we use it to automatically generate the synthetic counterpart of the @xmath4, nl} sections of must-cinema. the resulting data is respectively used to train two subst systems. the goal is to achieve comparable performance to that of similar models trained on manually segmented subtitles. for this purpose, using the same architecture, we also train two systems on the original manual segmentations of must-cinema. as shown in ## conclusions we presented an automatic segmenter able to turn existing st corpora into subst training data. through comparative experiments on two language pairs in zero-shot conditions, we showed that subst systems trained on this synthetic data are competitive with those built on human-annotated subtitling corpora. building on these positive results, and conditioned to the availability of suitable benchmarks, verifying the portability of the approach to a larger set of languages and domains is our priority for future work. the textual segmenter is a transformer-based @xcite architecture consisting of 3 encoder layers and 3 decoder layers. we set the hyper-parameters as in the fairseq @xcite multilingual translation task, both for the mono-and multilingual textual segmenters. for the multilingual model, a mini-batch for each language direction (here: 4) is built and the model weights are updated after each mini-batch, a mechanism already present in fairseq multilingual machine translation @xcite . the multimodal segmenter is an extension of the textual segmenter encoder-decoder structure with an additional speech encoder composed of 12 transformer encoder layers as in the original speech-to-text task @xcite but with the addition of a ctc @xcite module to avoid the speech encoder pre-training @xcite . the training of the multilingual models is performed by pre-pending the language token (en, de, fr, it) to the target sentence @xcite , a mechanism already present in the fairseq speech-to-text library @xcite . the encoder and decoder embeddings are shared. we select the hyper-parameters of the original implementation, 9 except for a higher learning rate of 1•10 -3 , since pre-training was skipped. the vocabulary is generated using sentencepiece @xcite , setting the size to 10k unigrams both for the mono-and multilingual segmenters. for the supervised baseline using opensubtitles data, we follow the data selection process for the highest-performing segmenter in @xcite . we first filter sentences with subtitles of maximum 42 characters. since line breaks are not present in opensubtitles, we @xmath6.25, paying attention not to insert two @xmath7. this proportion reflects @xmath8. we noted that almost 90% of the sentences filtered contain only one subtitle. this is not very informative for the segmenter, since the only operation required is inserting @xmath9. for this reason, we further select only sentences with at least two subtitles (or two subtitle lines). this results in 2,956,207 sentences for es and 683,382 sentences for nl. we then add the same number of 9 https://github.com/pytorch/fairseq/ blob/main/examples/speech_to_text/docs/ mustc_example.md sentences containing only one subtitle. after this process, we obtain 5,912,414 sentences for es and 1,366,764 sentences for nl. the supervised baseline is trained with the same settings as the textual monolingual segmenter. for the count chars baseline, a break is inserted before reaching the 42-character limit, as per ted guidelines. if the 42-character limit is reached in the middle of a word, the break is inserted before this word. this method will always obtain a 100% conformity to the length constraint. as with the data filtering process, <@xmath10.25. for the subst models discussed in section 5, we use the speech-to-text task small architecture of fairseq with the additional ctc module as in @xcite . we use 4 gpus k80 for training all the architectures: it takes around 1 day for the textual-only and around 1 week for the multimodal segmenters and the subst models. all results are obtained by averaging 7 checkpoints (best, three preceding and three succeeding checkpoints).