Video-Helpful Multimodal Machine Translation (2023)

Abstract: Existing multimodal machine translation (MMT) datasets consist of images and video captions or instructional video subtitles, which rarely contain linguistic ambiguity, making visual information ineffective in generating appropriate translations. Recent work has constructed an ambiguous subtitles dataset to alleviate this problem but is still limited to the problem that videos do not necessarily contribute to disambiguation. We introduce EVA (Extensive training set and Video-helpful evaluation set for Ambiguous subtitles translation), an MMT dataset containing 852k Japanese-English parallel subtitle pairs, 520k Chinese-English parallel subtitle pairs, and corresponding video clips collected from movies and TV episodes. In addition to the extensive training set, EVA contains a video-helpful evaluation set in which subtitles are ambiguous, and videos are guaranteed helpful for disambiguation. Furthermore, we propose SAFA, an MMT model based on the Selective Attention model with two novel methods: Frame attention loss and Ambiguity augmentation, aiming to use videos in EVA for disambiguation fully. Experiments on EVA show that visual information and the proposed methods can boost translation performance, and our model performs significantly better than existing MMT models.

## introduction neural machine translation (nmt) @xcite ) models relying on text data have achieved state-of-the-art performance. however, in many cases, the text is insufficient to provide the information needed for appropriate translation, especially when the source text is ambiguous. in this work, "ambiguous" refers not only to ambiguity in a narrow sense caused by target . . . factors such as polysemy but also to different possible translations caused by factors such as emotion, politeness, and omission that needs multimodal information for disambiguation. for a source text, if there are multiple possible translations and some of them are more appropriate than others under certain visual scenes, we call it ambiguous. multimodal machine translation (mmt) @xcite uses visual data as auxiliary information to tackle the ambiguity problem. the contextual information in the visual data helps to resolve the ambiguity in the source text data. previous mmt studies have mainly focused on the image-guided machine translation (imt) task @xcite @xcite , where, given an image and a source sentence, the goal is to enhance the quality of translation by leveraging their semantic correspondence to the image. resolving ambiguities through visual cues is one of the main motivations behind this task. compared with images, videos contain ordered sequences of frames and can provide richer visual features such as motion features. recently, some studies have started to focus on the video-guided machine translation (vmt) task @xcite . vmt faces the problem of data scarcity. the how2 @xcite and vatex @xcite datasets are recent efforts to allevi-ate the problem. in addition, previous datasets are limited to subtitles of instructional videos or video captions that describe the video clips. it has been shown that caption mmt essentially does not require visual information due to the lack of language ambiguity in captions @xcite . at the same time, the subtitles of the instructional videos are similar to captions and also lack ambiguity. the visa @xcite dataset has made efforts to solve this problem. visa contains parallel subtitles and corresponding video clips collected from movies or tv episodes in which the source subtitles are ambiguous. however, it has a limitation that the videos do not necessarily contribute to disambiguation. to address the problems of previous vmt datasets, we construct a new large-scale vmt dataset eva (extensive training set and videohelpful evaluation set for ambiguous subtitles translation) for vmt research. eva has an extensive training set, and a video-helpful evaluation set in which source subtitles are ambiguous and videos are guaranteed to be helpful for disambiguation. in total, eva contains 852k ja-en parallel subtitle pairs, 520k zh-en parallel subtitle pairs, and corresponding video clips collected from movies and tv episodes, where each pair of parallel subtitles has a corresponding video clip. subtitles from movies and tv episodes are essentially dialogues and short (7.34 english words in our case), which makes subtitles have many possible interpretations and thus makes videos helpful. to train a vmt model that can disambiguate translations, it is necessary for the training set to contain possible translation patterns. to achieve this, a simple yet effective way is to make the training set as large as possible. the training set of eva contains 848k ja-en parallel subtitle pairs, 517k zh-en parallel subtitle pairs, and corresponding video clips, which are collected from 763 movies and 1, 361 tv episodes with a total length of 3, 791 hours. the training set is much larger than existing vmt datasets and may cover more subtitle translation patterns. for testing vmt models, it is inappropriate to use general data because many translations do not require visual information. therefore, we select video-helpful data to construct an evaluation set that contains 4, 276 ja-en parallel subtitle pairs, 2, 940 zh-en parallel subtitle pairs, and corresponding video clips. considering the definition of am-biguity and the need for the training set containing possible translation patterns, we collect videohelpful data using translation sets, which are sets of parallel subtitles that have the same source subtitles but different target subtitles. an example of a translation set is shown in figure 1 . each pair of parallel subtitles belongs to a video clip. as a translation task, the video clip can help us translate the source subtitle. the first video clip shows two women escaping, suggesting a "go" translation, while the last video clip portrays a scene involving an army, suggesting a "forward" translation. we construct the dataset by collecting parallel subtitles and corresponding video clips, constructing an evaluation set with translation sets and crowdsourcing, and then using the remaining part as the training set. furthermore, we propose an mmt model safa (selective attention model with frame attention loss and ambiguity augmentation) for the vmt task. based on the selective attention model proposed @xcite for the imt task, we 1) use clip4clip @xcite model to extract video features, 2) propose frame attention loss to make the model focus more on the central frames where the subtitles occur, and 3) propose ambiguity augmentation to make the model put more weights on the possibly-ambiguous data. experiments on eva show that safa achieves 15.41 bleu score and 35.86 meteor score for ja-en translation, and 27.62 bleu score and 48.74 meteor score for zh-en translation, which are significantly better than existing mmt models. furthermore, our proposed methods significantly improve mt performance when incorporating videos, with relative improvements of 9.99% and 4.94% in bleu scores for ja-en and zh-en translations, respectively. in summary, our contributions are three-fold: ## dataset in this section, we outline the construction of eva, which is a vmt dataset combined with parallel subtitles and corresponding video clips. the dataset contains a large-scale training set and a small videohelpful evaluation set. for the former, we make it large enough to cover different translation patterns. for the latter, we ensure that the source subtitles are ambiguous and the video clips can help disambiguate the source subtitles. ## model we present a new model, safa, for vmt based on the eva dataset. previous vmt models mainly focus on caption translation based on the existing video caption datasets. in contrast, our model mainly focuses on subtitle translation. we design safa based on a selective attention model, propose a frame attention loss to make the model focus on the central frames where the subtitles occur, and use ambiguity augmentation to make the model put more weight on the possibly-ambiguous data. the model overview is shown in figure 4 . ## conclusion the paper introduced a new vmt dataset called eva, which contains an extensive training set and a video-helpful evaluation set in which videos are guaranteed to be helpful for disambiguating source subtitles. in addition, we proposed a novel vmt model called safa that incorporates selective attention with frame attention loss and ambiguity augmentation. experiments on eva demonstrated that visual information and the proposed methods can boost translation performance, and safa performs significantly better than previous vmt models. we hope that this work will inspire further research in this field.