Between Flexibility and Consistency: Joint Generation of Captions and Subtitles (2021)

Abstract: Speech translation (ST) has lately received growing interest for the generation of subtitles without the need for an intermediate source language transcription and timing (i.e. captions). However, the joint generation of source captions and target subtitles does not only bring potential output quality advantages when the two decoding processes inform each other, but it is also often required in multilingual scenarios. In this work, we focus on ST models which generate consistent captions-subtitles in terms of structure and lexical content. We further introduce new metrics for evaluating subtitling consistency. Our findings show that joint decoding leads to increased performance and consistency between the generated captions and subtitles while still allowing for sufficient flexibility to produce subtitles conforming to language-specific needs and norms.

## introduction new trends in media localisation call for the rapid generation of subtitles for vast amounts of audiovisual content. speech translation, and especially direct approaches @xcite , have recently shown promising results with high efficiency because they do not require a transcription (manual or automatic) of the source speech but generate the target language subtitles directly from the audio. however, obtaining the intralingual subtitles (hereafter "captions") is necessary for a range of applications, while in some settings captions need to be displayed along with the target language subtitles. such "bilingual subtitles" are useful in multilingual online meetings, in countries with multiple official languages, or for language learners and audiences with different accessibility needs. in those cases, captions and subtitles should not only be consistent with the visual and acoustic dimension of the audiovisual material but also between each other, for example in the number of blocks (pieces of time-aligned text) they occupy, their length and segmentation. consistency is vital for user experience, for example in order to elicit the same reaction among multilingual audiences, or to facilitate the quality assurance process in the localisation industry. previous work in st for subtitling has focused on generating interlingual subtitles @xcite , a) without considering the necessity of obtaining captions consistent with the target subtitles, and b) without examining whether the joint generation leads to improvements in quality. we hypothesise that knowledge sharing between the tasks of transcription and translation could lead to such improvements. moreover, joint generation with a single system can avoid the maintenance of two different models, increase efficiency, and in turn speed up the localisation process. lastly, if joint generation improves consistency, joint models could increase automation in subtitling applications where consistency is a desideratum. in this work, we address these issues for the first time, by jointly generating both captions and subtitles for the same audio source. we experiment with the following models: 1) shared direct @xcite , where the speech encoder is shared between the transcription and the translation decoder, 2) two-stage @xcite , where the transcript decoder states are passed to the translation decoder, and 3) triangle @xcite , which extends the two-stage by adding a second attention mechanism to the translation decoder which attends to encoded speech inputs. we compare these models with the established approaches in st for subtitling: an independent direct st model and a cascade (asr+mt) model. moreover, we extend the evaluation beyond the usual metrics used to assess transcription and translation quality (respectively wer and bleu), by also evaluating the form and consistency of the generated subtitles. @xcite introduced several lexical and surface metrics to measure consistency of st outputs, but they were only applied to standard, non-subtitle, texts. subtitles, however, are a particular type of text structured in blocks which accompany the action on screen. therefore, we propose to measure their consistency by taking advantage of this structure and introduce metrics able to reward subtitles that share similar structure and content. our contributions can be summarised as follows: ## analysis 6.1 evaluation of lexical consistency in this section, we test the reliability of the lexical consistency metric. the metric depends on the successful word alignment, which, especially for low quality text, might be sub-optimal. we therefore manually count the number of words in the subtitles which do not appear in the corresponding captions. the task is performed on the first 347 sentences of the output of dirmu for french and german. we then estimate the mean absolute error between the consistency metric computed using the manual and the automatic alignments. as an additional step, we compute how often the automatic and the manual annotations agree in their judgement of consistent/non-consistent content in each block. the mean absolute error between the manually and the automatically computed score is .08 for french and .11 for german. the metric may not be able to account for very small score differences between systems, however, when inspecting the differences between manual and automatic annotation we noticed that most errors appear in very low quality outputs or where lexical content was missing, and lead to a misalignment of only a few words. these cases were in fact challenging even for the human annotator. instead, the agreement in the consistent/not-consistent judgement is high, with .85 for french and .75 for german. considering the difficulty of aligning sentences belonging to languages with different word ordering, and the lower quality of german outputs, it is not surprising that the word aligner from english to german affects more our metric. however, these results show that the real impact is moderate and the metric is consistent with the human judgements in the majority of cases. ## conclusions in this work we explored joint generation of captions and subtitles as a way to increase efficiency and consistency in scenarios where this property is a desideratum. to this aim, we proposed metrics for evaluating subtitling consistency, tailored to the structural peculiarities of this type of translation. we found that coupled inference, either by models supporting end-to-end training (2st, tri) or not (cas), leads to quality and consistency improvements, but with a slight degradation of the conformity to target subtitle constraints. the final architectural choice depends on the flexibility versus conformity requirements of the application scenario. the findings of this work have provided initial insights related to the joint generation of captions and subtitles. one future research direction is towards improving the quality of generation by using more recent, higher-performing st architectures. for example, @xcite extended the notion of the dual decoder by adding an interactive attention mechanism which allows the two decoders to exchange information and learn from each other, while synchronously generating transcription and translation. @xcite proposed two variants of the dual decoder of @xcite , the cross and parallel dual decoder, and experimented with multilingual st. while neither of these works reported results on consistency, we expect that they are relevant to our scenario and have the potential of jointly generating multiple language/accessibility versions with high consistency. moving beyond generic architectures, in the future we are planning to experiment with tailored architectures for improving consistency between automatically generated captions and subtitles. one important insight emerging from this work is that different degrees of conformity are required, or even appropriate, depending on the application scenario and languages involved. given these challenges, we are aiming at developing approaches which allow for tuning the output to the desired degree of conformity, whether lexical, structural or both. we hope that this work will contribute to the line of research efforts towards improving efficiency and quality of automatically generated captions and subtitles.