Joint Generation of Captions and Subtitles with Dual Decoding (2022)

Abstract: As the amount of audio-visual content increases, the need to develop automatic captioning and subtitling solutions to match the expectations of a growing international audience appears as the only viable way to boost throughput and lower the related post-production costs. Automatic captioning and subtitling often need to be tightly intertwined to achieve an appropriate level of consistency and synchronization with each other and with the video signal. In this work, we assess a dual decoding scheme to achieve a strong coupling between these two tasks and show how adequacy and consistency are increased, with virtually no additional cost in terms of model size and training complexity.

## introduction as the amount of online audio-visual content continues to grow, the need for captions and subtitles foot_0 in multiple languages also steadily increases, as it widens the potential audience of these contents. t s t s c t s c t c,s a. b. c. d. both activities are closely related: human subtitle translators often generate subtitles directly based on the original captions without viewing or listening to the original audio/video file. this strategy however runs the risk of amplifying, in the subtitle approximations, simplifications or errors present in the captioning. it may even happen that both texts need to be simultaneously displayed on screen: for instance, in countries with several official languages, or to help foreign language learners. this means that captions and subtitles need to be consistent not only with the video content, but also with each other. it also implies that they should be synchronized @xcite . finally, even in scenarios where only subtitles would be needed, generating captions at the same time may still help to better check the correctness of subtitles. early approaches to automatic subtitling (e.g. @xcite also assumed a pipeline architecture (figure 1 (b) ), where subtitles are translated from captions derived from automatic speech transcripts. a recent alternative (figure 1 (a) ), which mitigates cascading errors, is to independently perform captioning and subtitling in an endto-end manner @xcite ; the risk however is to generate inconsistencies (both in alignment and content) between the two textual streams. this approach might also be limited by the lack of appropriate training resources @xcite . various ways to further strengthen the interactions between these tasks by sharing parameters or loss terms are evaluated by @xcite . figure 1 (c ) illustrates these approaches. in this work, we explore an even tighter integration consisting of simultaneously generating both captions and subtitles from automatic speech recognition (asr) transcripts using one single dual decoding process @xcite @xcite @xcite , illustrated in figure 1 (d ). generally speak- transcript i 'm combining specific types of signals the mimic how our body response to in an injury to help us regenerate caption i'm combining specific types of signals [eob] that mimic how our body responds to injury [eol] to help us regenerate. [eob] subtitle je combine différents types de signaux [eob] qui imitent la réponse du corps [eol] aux blessures pour nous aider à guérir. [eob] the transcript-to-subtitle task involves the same transformations, with an additional translation step to produce text in another language. as our experiments suggest, a tighter integration not only improves the quality and the consistency of captions and subtitles, but it also enables a better use of all available data, with hardly any impact on model size or training complexity. our main contributions are the following: (i) we show that simultaneously generating captions and subtitles can improve performance in both languages, reporting significant improvements in bleu score with respect to several baselines; (ii) we initialize dual decoder from a standard encoder-decoder model trained with large scale data, thereby mitigating the data scarcity problem; (iii) we explore a new parameter sharing scheme, where the two decoders share all their parameters, and achieve comparable performance at a much reduced model size in our experimental conditions; (iv) using 2-round decoding, we show how to alleviate the exposure bias problem observed in dual decoding, leading to a clear boost in performance. ## conclusion in this paper, we have explored dual decoding to jointly generate captions and subtitles from asr transcripts. experimentally, we found that dual decoding improves translation quality for both captioning and subtitling, while delivering more con-sistent output pairs. additionally, we showed that (a) model sharing on the decoder side is viable and effective, at least for related languages; (b) initializing with pre-trained models vastly improves performance; (c) 2-round decoding allowed us to mitigate the exposure bias problem in our model. in the future, we would like to experiment on more distant language pairs to validate our approach in a more general scenario.