B ig V ideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation (2023)

Abstract: We present a large-scale video subtitle translation dataset, *BigVideo*, to facilitate the study of multi-modality machine translation. Compared with the widely used *How2* and *VaTeX* datasets, *BigVideo* is more than 10 times larger, consisting of 4.5 million sentence pairs and 9,981 hours of videos. We also introduce two deliberately designed test sets to verify the necessity of visual information: *Ambiguous* with the presence of ambiguous words, and *Unambiguous* in which the text context is self-contained for translation. To better model the common semantics shared across texts and videos, we introduce a contrastive learning method in the cross-modal encoder. Extensive experiments on the *BigVideo* shows that: a) Visual information consistently improves the NMT model in terms of BLEU, BLEURT and COMET on both Ambiguous and Unambiguous test sets. b) Visual information helps disambiguation, compared to the strong text baseline on terminology-targeted scores and human evaluation.

## introduction humans are able to integrate both language and visual context to understand the world. from the perspective of nmt, it is also much needed to make use of such information to approach humanlevel translation abilities. to facilitate multimodal machine translation (mmt) research, a number of datasets have been proposed including imageguided translation datasets @xcite @xcite and videoguided translation datasets @xcite @xcite . however, the conclusion about the effects of visual information is still unclear for mmt research @xcite . previous work has suggested that visual information is only marginally beneficial for machine translation @xcite , especially when the text context is not complete. the most possible reason is that existing datasets focus on captions describing images or videos, which are not large and diverse enough. the text inputs are often simple and sufficient for translation tasks @xcite . take the widely used multi30k as an example. multi30k consists of only 30k image captions, while typical text translation systems are often trained with several million sentence pairs. we argue that studying the effects of visual contexts in machine translation requires a large-scale and diverse data set for training and a real-world and complex benchmark for testing. to this end, we propose bigvideo, a large-scale video subtitle translation dataset. we collect human-written subtitles from two famous online video platforms, xigua and youtube. bigvideo consists of 155 thousand videos and 4.5 million high-quality parallel sentences in english and chinese. we highlight the key features of bigvideo as follows: a) the size of bigvideo bypasses the largest available video machine translation dataset how2 and vatex by one order of magnitude. b) to investigate the need for visual information, two test sets are annotated by language experts, referred as ambiguous and unambiguous. in ambiguous, the source input is not sufficient enough and requires videos to disambiguate for translation. the experts also labelled unambiguous words to help evaluate whether the improvement comes from visual contexts. in un-ambiguous, actions or visual scenes in the videos are mentioned in the subtitles but source sentences are self-contained for translation. to make the most of visual information for mmt, we propose a unified encoder-decoder framework for mmt. the model has a cross-modal encoder that takes both videos and texts as inputs. motivated by recent work on cross-modal learning @xcite @xcite , we also introduce a contrastive learning objective to further bridge the representation gap between the text and video and project them in a shared space. as such, the visual information can potentially contribute more to the translation model. we conduct extensive experiments on the proposed benchmark bigvideo and report the results on bleu @xcite , bleurt @xcite , comet @xcite , terminology-targeted metrics and human evaluation. we also introduce the large scale wmt19 training data, which contains 20.4m parallel sentences to build the strong baseline model. the experiments show that visual contexts consistently improve the performance of both the ambiguous and unambiguous test set over the strong textonly model. the finding is slightly different with previous studies and address the importance of a large scale and high-quality video translation data. further, the contrastive learning method can further boost the translation performance over other visual-guided models, which shows the benefits of closing the representation gap of texts and videos. to leverage video inputs in machine translation models, @xcite use pretrained models such as resnet @xcite , faster-rcnn @xcite and i3d @xcite . an additional attention module is designed in the rnn decoder to fuse visual features. to better learn temporal information in videos, @xcite propose a hierarchical attention network to model video-level features. different from previous work, we use a unified encoder to learn both video and text features. specifically, a contrastive learning objective is adopted to learn cross-modal interaction. image-guided machine translation. images as additional inputs have long been used for machine translation @xcite . for neural models, several attempts have been focused on enhancing the sequence-to-sequence model with strong image features @xcite @xcite @xcite @xcite @xcite . however, @xcite and @xcite point out that images in multi30k provide little information for translation. in this work, we focus on videos as additional visual inputs for subtitle translation. videos illustrate objects, actions, and scenes, which contain more information compared to images. subtitles are often in spoken language, which contains inherent ambiguities due to multiple potential interpretations @xcite . hence, our dataset can be a complement to existing mmt datasets. dataset text video # sent len. # video # clip sec. visa 35k 7.0 2k 35k 10.0 vatex 129k 15.2 25k 25k 10.0 how2 186k 20.6 13k 186k 5.8 bigvideo 4.5m 22.8 156k 4.5m 8.0 ## dataset we present bigvideo, consisting of 150 thousand unique videos (9,981 hours in total) with both english and chinese subtitles. the videos are collected from two popular online video platforms, youtube and xigua. all subtitles are humanwritten. ## conclusion in this paper, we present bigvideo --a largescale video subtitle translation dataset for multimodal machine translation. we collect 155 thousand videos accompanied by over 4.5 million bilingual subtitles. specially, we annotate two test subsets: ambiguous where videos are required for disambiguation and unambiguous where text contents are self-contained for translation. we also propose a cross-modal encoder enhanced with a contrastive learning objective to build cross-modal interaction for machine translation. experimental results prove that videos consistently improve the nmt model in terms of the translation evaluation metrics and terminology-targeted metrics. moreover, human annotators prefer our system outputs, compared to the strong text-only baseline. we hope our bigvideo dataset can facilitate the research of multi-modal machine translation.