Feature-level Incongruence Reduction for Multimodal Translation (2021)

Abstract: Caption translation aims to translate image annotations (captions for short). Recently, Multimodal Neural Machine Translation (MNMT) has been explored as the essential solution. Besides of linguistic features in captions, MNMT allows visual(image) features to be used. The integration of multimodal features reinforces the semantic representation and considerably improves translation performance. However, MNMT suffers from the incongruence between visual and linguistic features. To overcome the problem, we propose to extend MNMT architecture with a harmonization network, which harmonizes multimodal features(linguistic and visual features)by unidirectional modal space conversion. It enables multimodal translation to be carried out in a seemingly monomodal translation pipeline. We experiment on the golden Multi30k-16 and 17. Experimental results show that, compared to the baseline,the proposed method yields the improvements of 2.2% BLEU for the scenario of translating English captions into German (En→De) at best,7.6% for the case of English-to-French translation(En→Fr) and 1.5% for English-to-Czech(En→Cz). The utilization of harmonization network leads to the competitive performance to the-state-of-the-art.

## introduction caption translation is required to translate a sourcelanguage caption into target-language, where a caption refers to the sentence-level text annotation of an image. as defined in the shared multimodal translation task 1 in wmt, caption translation can be conducted over both visual features in images and linguistic features of the accompanying captions. the question of how to opportunely utilize images for caption translation motivates the study of multimodality, including not only the extraction of visual features but the cooperation between visual and linguistic features. in this paper, we follow 1 http://www.statmt.org/wmt16/ the previous work @xcite to boil caption translation down to a problem of multimodal machine translation. so far, a large majority of previous studies tend to develop a neural network based multimodal machine translation model (viz., mnmt), which consists of three basic components: multimodal learning networks have been successfully grounded with different parts of various neural translation networks. they are proven effective in enhancing translation performance. nevertheless, the networks suffer from incongruence between visual and linguistic features because: considering the figure 1 ("counterfeit" means image captioning output), the visual features enable a image processing model to recognize "two horses" as well as their position relative to a "sleigh". however, such features are obscure for a translation model and useful for translating a verb, such as "pulling" in the caption. in this case, incongruence of heterogeneous features results from the unawareness of the correspondence between spatial relationship ("running horses" ahead of "sleigh") and linguistic semantics ("pulling"). to ease the incongruence, we propose to equip the current mnmt with a harmonization network, in which visual features are not directly introduced into the encoding of linguistic semantics. instead, they are transformed into linguistic features before absorbed into semantic representations. in other words, we tend to make a detour during the cross-modality understanding, so as to bypass the modality barrier (figure 2 ). in our experiments, we employ a captioning model to conduct harmonization. the hidden states it produced for decoding caption words are intercepted and involved into the representation learning process of mnmt. the rest of the paper is organized as follows: section 2 presents the motivation and methodological framework. section 3 gives the nmt model we use. in section 4, we introduce the captioning model that is trainable for cross-modality feature space transformation. section 5 presents the captioning based harmonization networks as well as the resultant mnmt models. we discuss test results in section 6 and overview the related work in section 7. we conclude the paper in section 8. ## fundamentals and methodological framework we utilize @xcite 's image captioning (cap for short) to guide the cross-modality feature transformation, converting visual features into linguistic. cap is one of the generation models which are specially trained to generate language conditioned on visual features of images. ideally, during training, it learns to perceive the correspondence between visual and linguistic features, such as that between the spatial relationship of "running dogs ahead of a sled" in figure 1 and the meaning of the verb "pulling". this allows cap to produce appropriate linguistic features during testing in terms of similar visual features, such as that in the case of predicting the verb "pulling" for the scenario of "running horses ahead of a sleigh". methodologically speaking, we adopt the linguistic features produced by the encoder of cap instead of the captions generated by the decoder of cap. on the basis, we integrate both the linguistic features of the original source-language caption and those produced by cap into calixto et al (2017b)'s attention-based cross-modality learning model (see figure 3 ). experimenal results show that the learning model substantially improves bahdanau et al (2014)'s encoder-decoder nmt system. ## preliminary 1: attentive encoder-decoder nmt (baseline) bahdanau et al. (2014)'s attentive encoderdecoder nmt as the baseline. it is constructed gru gru xt-1 s(t-1) gru gru xt s(t) gru gru xt+1 s(t+1) gru gru xn s(n) gru gru x1 s(0) s(t) s(t+1) s(n) s(t-1) s(0) attention gru gru hd2(t) he1(n) he2(n) c(t) hd1(t) + gru gru hd2(t-1) c(t-1) hd1(t-1) + gru c(0) hd1(1) + hd2(1) gru gru hd2(t+1) c(t+1) hd1(t+1) + gru gru hd2(m) c(m) hd1(m) + attention attention attention attention ht(fnl) hd1/d2(0) gru y(t-2) y(t-1) y(t) y(m) he1(0) cap v nmt decoder nmt encoder cap v cap-att cap v cap-tkn y(t)=y(t)hd2(fnl) cap v cap-enc he1(0)=tanh(w0hd2(fnl)) cnn cnn cnn cap-dec hd1(0)=w' 0 hd2(fnl) hd2(0)=w' 0 hd2(fnl) image encoder modality transition cap-enc he2(0)=tanh(w0hd2(fnl)) cross-modality learning he2(0) he1(t) he2(n-t+1) cap gru gru xt s(t) attention gru gru hd2(t) c(t) hd1(t) + hd2(t-1) y(t-1) he1(t) he2(n-t+1) hd1(t) hd1(t-1) y(t-1) baseline nmt congruent cross-modality learning for multimodal nmt with a bigru encoder and a conditional gru (cgru) decoder @xcite . attention mechanism is used between bigru and cgru. the diagram at the right side of figure 3 shows the baseline framework. for a source-language caption, we represent it with a sequence of randomly-initialized (kalchbrenner and blunsom, 2013) word embeddings @xmath0, ..., x n ), where each x t is uniformly specified as a k-dimensional word embedding. conditioned on the embeddings, chung et al (2014)'s bigru is used to compute the bidirectional hidden states @xmath1, ..., s n ), where each s t is obtained by combining the t-th hidden state of forward gru and that of backward gru: s @xmath2, ← ---gru e (x t )]. padding @xcite and dynamic stabilization @xcite are used. firat and cho (2016)'s cgru is utilized for decoding, which comprises two forward gru units, i.e., ---→ gru d1 and ---→ gru d2 respectively. the prediction y t denotes the k-dimensional embedding of the predicted word at the t-th decoding step). by contrast, ---→ gru d2 serves to produce the attentive decoder hidden states h @xmath3, ..., h d2 m ), where each h d2 t is computed conditioned on the previous attentive state h d2 t-1 , the current inattentive state h d1 t , as well as the current attention-aware context c t : the context c t is obtained by the attention mechanism over the global encoder hidden states s: c @xmath4, where α t denotes the attention weight at the t-th time step. eventually, the prediction of each target-language word is carried out as follows (where, w h , w c , w y , b o and b y are trainable parameters): ## preliminary 2: image-dependent linguistic feature acquisition by cap for an image, captioning models serve to generate a sequence of natural language (caption) that describes the image. such kind of models are capable of transforming visual features into linguistic features by encoder-decoder networks. we utilize @xcite 's cap to obtain the transformed linguistic features. ## harmonization for mnmt in the previous work of multimodal nmt, visual features in v are directly used for cross-modality learning. by contrast, we transform visual features into image-dependent attention-aware linguistic features (i.e., second-layer hidden states ȟd2 t emitted by cap) before use. we provide four-class variants of cross-modality learning to improve nmt. they absorb image-dependent attention-aware linguistic features in different ways, including a variant that comprises attentive feature fusion (cap-att) and three variants (cap-enc, cap-dec and cap-tkn) which carry out reinitialization and target-language embedding modulation. figure 3 shows the positions in the baseline nmt where the variants come into play. cap-att intends to improve nmt by conducting joint representation learning across the features of the source-language caption and that of the accompanying image. on one side, cap-att adopts the encoder hidden state s t (emitted by the bi-gru encoder of the baseline nmt) and uses it as the language-dependent linguistic feature. on the other side, it takes the image-dependent attentionaware linguistic feature ȟd2 t (produced by cap). we suppose that the two kinds of features (i.e., ȟd2 t and s t ) are congruent with each other. on the basis, cap-att blends ȟd2 t into s t to form the joint representation s t . element-wise feature fusion @xcite ) is used to compute s t : s @xmath5. using the joint representation s t , cap-att updates the attention-aware context c t which is fed into the cgru decoder of the baseline nmt: c @xmath6, ∀ @xmath7. by substituting the updated context c t into the computation of the cgru decoder, cap-att further refines the decoder hidden state h d 2 t and prediction of target-language words. equation 2 formulates the decoding process, where d t is the shorthand of equation (1). ## experimentation 6.1 resource and experimental datasets we perform experiments on multi30k-16 and multi30k-17 foot_0 , which are provided by wmt for the shared tasks of multilingual captioning and multimodal mt @xcite . the corpora are used as the extended versions of flichr30k @xcite , since they contain not only english (en) image captions but their translations in german (de), french (fr) and czech (cz). hereinafter, we specify an example in multi30k as an image which is accompanied by three @xmath8, @xmath9. each of multi30k-16 and multi30k-17 contains about 31k examples. we experiment on the corpora separately, and as usual divide each of them into training, validation and test sets, at the scale of 29k, 1,014 and 1k examples, respectively. in addition, we carry out a complementary experiment on the ambiguous coco which contains 461 examples @xcite . due to the inclusion of ambiguous verbs, the examples in ambiguous coco can be used for the evaluation of visual sense disambiguation in a mnmt scenario. ## related work we have mentioned the previous work of mnmt in section 1, where the research interest has been classified into image encoding, encoder-decoder nmt construction and cross-modality learning. besides, we present the methods of @xcite and @xcite in the section 4.4.2, along with the systematic analysis. besides, many scholars within the research community have made great efforts upon the development of sophisticated nmt architectures, including multi-source @xcite , multi-task @xcite and multi-way @xcite nmt, as well as those equipped with attention mechanisms @xcite . the research activities are particularly crucial since they broaden the range of cross-modality learning strategies. current research interest has concentrated on the incorporation of visual features into nmt @xcite , by means of visual-linguistic context vector concatenation @xcite , doubly-attentive decoding @xcite , hierarchical attention combination @xcite , cross-attention network @xcite , gated attention network @xcite , joint @xcite and ensemble @xcite learning . in addition, image attention optimization @xcite and monolingual data expansion @xcite have been proven effective in this field. @xcite use an off-shelf object detector and an additional image dataset @xcite ## conclusion we demonstrate that the captioning based harmonization model reduces incongruence between multimodal features. this contributes to the performance improvement of mnmt. it is proven that our method increases the use efficiency of images. the interesting phenomenon we observed in the experiments is that modality incongruence reduction is more effective in the scenario of @xmath10. this raises a problem of adaptation to languges. in the future, we will study on the distinct grammatical and syntactic principles of target languages, as well as their influences on the adaptation. for example, the syntax of french can be considered as most strict. thus, a sequence-dependent feature vector may be more adaptive to mnmt towards french. accordingly, we will attempt to develop a generative adversarial network based adaptation enhancement model. the goal is to refine the generated linguistic features by learning to detect and eliminate the features of less adaptability.