Video-guided Machine Translation with Spatial Hierarchical Attention Network (2021)

Abstract: Video-guided machine translation, as one type of multimodal machine translations, aims to engage video contents as auxiliary information to address the word sense ambiguity problem in machine translation. Previous studies only use features from pretrained action detection models as motion representations of the video to solve the verb sense ambiguity, leaving the noun sense ambiguity a problem. To address this problem, we propose a video-guided machine translation system by using both spatial and motion representations in videos. For spatial features, we propose a hierarchical attention network to model the spatial information from object-level to video-level. Experiments on the VATEX dataset show that our system achieves 35.86 BLEU-4 score, which is 0.51 score higher than the single model of the SOTA method.

## introduction neural machine translation (nmt) models relying on text data @xcite have achieved high performance for domains where there is less ambiguity in data such as the newspaper domain. for some other domains, especially real-time domains such as spoken language or sports commentary, the verb and the noun sense ambiguity largely affects the translation quality. to solve the ambiguity problem, multimodal machine translation (mmt) @xcite focuses on incorporating visual data as auxiliary information, where the spatiotemporal contextual information in the visual data helps reduce the ambiguity of nouns or verbs in the source text data @xcite . previous mmt studies mainly focus on imageguided machine translation (imt) task @xcite . however, videos are better information sources than images because one source: an apple picker takes apples from the trees and places them in a bin. translation: ⼀ 个 苹 果 苹 果 从 树 上 摘 下 苹 果 ， 然 后 把 它 们 放 在 ⼀ 个 垃 圾 桶 ⾥ 。( an apple apple takes apples from the trees and places them in a trash bin.) figure 1 : an example with the noun sense ambiguity problem in the vmt model by @xcite . the research of video-guided machine translation (vmt) starts from a large-scale video-andlanguage-research dataset (vatex) @xcite . the authors also established a baseline using features from pretrained action detection models as motion representations of the video, which addresses the verb sense ambiguity to some extent, leaving noun sense ambiguity unsolved. @xcite aims to solve both the verb and noun sense ambiguity problems by using frame-level action, object, and scene representations. however, without using detailed spatial information within one frame and contextual information between frames, the effect of resolving the noun ambiguity problem is limited. for example, as shown in figure 1 , the noun "bin" in english is wrongly translated into "trash bin" in chinese, which should be translated into "box." in this work, we propose a vmt system to address both the verb and the noun sense ambiguity problems by using both motion and spatial representations in a video. to obtain spatial representations efficiently, we propose to use a hierarchical attention network (han) @xcite to model the spatial information from object-level to video-level, thus we call it the spatial han module. additionally, to obtain a better contextual spatial information, we add several kinds of middle layers between the object-to-frame layer and frameto-video layer in the original han. experiments on the vatex dataset @xcite show that our vmt system achieves 35.86 corpus-level bleu-4 score on the vatex test set, yielding a 0.51 score improvement over the single model of the sota method @xcite . ## vmt with spatial han the overview of the proposed model is presented in figure 2 , which consists of components in the vmt baseline model @xcite and our proposed spatial han module. ## conclusion in this work, we proposed a vmt system with spatial han, which achieved 0.51 bleu score improvement over the single model of the sota method. the result also showed the effectiveness of spatial features for the noun sense disambiguation. our future work will focus on the alignment between text, motion and spatial representations.