Entity-level Cross-modal Learning Improves Multi-modal Machine Translation (2021)

Abstract: Multi-modal machine translation (MMT) aims at improving translation performance by incorporating visual information. Most of the studies leverage the visual information through integrating the global image features as auxiliary input or decoding by attending to relevant local regions of the image. However, this kind of usage of visual information makes it difficult to figure out how the visual modality helps and why it works. Inspired by the findings of (CITATION) that entities are most informative in the image, we propose an explicit entity-level cross-modal learning approach that aims to augment the entity representation. Specifically, the approach is framed as a reconstruction task that reconstructs the original textural input from multi-modal input in which entities are replaced with visual features. Then, a multi-task framework is employed to combine the translation task and the reconstruction task to make full use of cross-modal entity representation learning. The extensive experiments demonstrate that our approach can achieve comparable or even better performance than state-of-the-art models. Furthermore, our in-depth analysis shows how visual information improves translation.

## introduction multi-modal machine translation (mmt) aims at improving the translation performance with the help of visual information such as image @xcite @xcite . the assumption behind this is that images consist of relatively complete information compared with textual description and can provide complementary knowledge to guide translation @xcite . previous studies mainly focus on integrating the visual information into neural machine translation as a global feature or as attention-based local features. benefiting from the similar representation between visual features @xcite and textual hidden states @xcite , several attempts have been made to incorporate image features as an auxiliary input to exploit its global semantics @xcite @xcite . some works leverage the spatial information in the decoding stage by attending to relevant local regions of the image @xcite @xcite @xcite @xcite . however, these sentence-level approaches which implicitly incorporate image features make it extremely difficult to figure out how visual features affect the representation of source-side sentences or the decision when generating a target-side word. furthermore, results from @xcite have shown that visual information maybe not the reason why mmt models were promoted, and it is observed that irrelevant images can improve translation unexpectedly. inspired by the findings of @xcite that entities are most informative in the image, we propose an entity-level cross-modal learning approach for multi-modal machine translation (emmt). different from sentence-level crossmodal semantics fusion approaches, our approach aims to augment the entity representation explicitly. we frame the entity-level cross-modal learning approach as a reconstruction task that reconstructs the original textual sentence from a degraded multimodal input @xcite . the multi-modal input is a mixture of a degraded sentence and related visual objects. the degraded sentence is generated by erasing the visually depictable entity words as done by @xcite and filling the erased position with corresponding visual objects. reconstructed from this kind of input, entity words are learned in a cross-modal way. then, a multi-task framework is employed to combine the translation task and the reconstruction task. thanks to the shared parameters from the reconstruction model, the translation could make full use of crossmodal entity representation learning and significant gains are obtained. we further take an in-depth analysis to figure out why the approach works by contrasting the translation correctness of entity words with several mmt models. the results show that the translation accuracy of entity words significantly increases with the help of visual information. the major contributions of our work are listed as follows: ## our approach in this section, we first introduce how our entitylevel cross-modal learning approach explicitly incorporates visual information into entity words. then, we frame this approach as a reconstruction task and combine it with the translation task in a multi-task framework. finally, we provide three parameter-sharing schemes to fully exploit the advantage of this multi-task learning approach. ## experimental setup we test our approach on both rnn-based and transformer-based models and carry out experiments on english to german (@xmath0. dataset we test our approach on the multi30k dataset @xcite in which each image is paired with one english description and one translated german description. multi30k was split into three parts: training, validation, and test, containing 29,000, 1,014, and 1,000 pairs of sentences respectively. we also evaluate our model in the multi30k entity extraction to extract visually depictable phrases and detect the corresponding visual objects, we apply an approach similar to the work @xcite . first, we employ an advanced natural language processing toolkit spacy to extract noun phrases in the source-side sentences. for wordlevel replacement mentioned in subsection 2.1, we keep the nouns in a phrase as the entity word. this affects 32.6% of the words in both the training and the test set for word-level replacement and 45.1% for phrase-level replacement. we measure the medians of entity word frequency in the word-level and the phrase-level replacement. both of the medians are 2 which means most of the entity words are low frequency. then, we employ the visual grounding toolkit released by @xcite to detect the visual objects which are related to the extracted noun phrases. theoretically, only visually depictable phrases are detectable. the ground truth visual bounding boxes and the entity phrases are given by flickr30k entities dataset @xcite . finally, we apply the resnet-50 @xcite pre-trained on imagenet @xcite to extract 2,048d global features for the visual objects. ## entity word analysis in this section, we take an in-depth analysis to find out why our entity-level cross-modal learning approach works. we intuitively assume that the approach provides an extra gain to the translation correctness of entity words. therefore, we measure the translation accuracy for different types of words and subtract the result of the baseline model from mmt models as the extra gain which we call the increment. we split all words into two parts: the entity words which were mentioned as word entity in subsection 2.1 and the other words which correspond to no visual object. different from sentence-level approaches, our entitylevel mmt models are expected to obtain more increment for the entity words. it is represented as lowering the increment difference between the other words and the entity words. the measurement is based on the sentence-level translation results of various mmt models. to get the word-level translation, we employ the fastalign @xcite toolkit which aligns tokens from source-side to target-side and concatenates the training set and the test set to train better alignments. the aligned target-side words are considered to be the translation of the source-side words. we take the alignment results of reference parallel data as the correct translation and compare it with the results of translated data from the mmt models. we pose a contrast among four kinds of mmt models: 6 of our rnn-based models, 12 of adversarial models in figure 3 : experiment results of entity word analysis. "all" words in the multi30k 2016 test are split into two parts. the "entity" represents the entity words in the word-level replacement rule. the "other" represents all other words. our models are in the red bounding boxes, and former mmt models are in the green bounding boxes. results on the detected visual objects the results of models "vo" are shown in figure 3 with red bounding boxes. a notable aspect of this figure is that most of the differences are positive. as mentioned in subsection 3, entity words are lowfrequency which makes it harder to learn a better representation. we sort all results in ascending order, based on their differences. it shows that all our models come out among the lowest differences which indicates image information helps to narrow the gap of translation quality between entity words and other words. results of our multi-task models as shown in figure 3 , neither adversarial models ("ro" and "rw") nor mlm models get stable lower differences. no evidence was found that the de-noising ability or the mlm of our multi-task scheme was a guarantee for helping the translations of entity words. it further proves that our entity-level cross-modal learning approach learns valuable visual information from visual objects. ## related work previous studies mainly focused on fusing the multi-modal information into the sentence-level semantics @xcite @xcite @xcite in the rnn-based architecture @xcite . besides above approaches, @xcite proposed to apply latent variables as the unified semantic representations. @xcite proposed a translate-and-refine approach to generate a good translation from the first draft by making better use of the target language and visual context. there are also works @xcite show that extra modality information is useful in a more fine-grained way. recently, @xcite proposed a finegrained method that employs a graph-based multimodal fusion encoder to fuse image and source text in the entity level. the input sentence and image are represented as a unified graph. the encoder can capture the relations among visual objects and linguistic entities. similar to their model, our model also explores the fine-grained multi-modal semantics at the entity-level. however, the differences lie in two aspects: (1) our entity-level cross-modal learning task is framed as a reconstruction problem that is simpler than their graph-based model, and we also investigate word-level and phrase-level entities. (2) our model does not rely on images during the inference stage. furthermore, our results are comparable or better than theirs. our work is mainly inspired by the image probing work @xcite . this work de-grades the textual input by replacing some depictable words with visual information and then tests whether the image is helpful. the authors find that the entities are most informative in the image. in our work, we utilize a similar degradation method to prepare the multi-modal input. we go a step further to learn an entity-level cross-modal representation that is proved to significantly improve translation performance. @xcite proposed several loss functions that help their model to capture more relevant information from visual objects. they also tried to weaken the functionality of textual modality to exploit visual information. the differences are as follows: (1) their approach needs entity labels manually annotated from the flickr30k entities dataset. (2) their model works well in the textual degradation situation. however, this strategy is not able to facilitate their model in the general intacttextual scenarios. @xcite also applied a multitask framework to perform multi-modal translation. the differences are in the following aspects: (1) their work fuse visual information in the sentence-level semantics. our models benefit from entity-level cross-modal fusion. (2) the image is grounded from source sentences in their model. in our approach, the model performs an imageto-text reconstruction task. (3) benefiting from the similar frameworks of the reconstruction task and the translation task, our multi-task method is flexible enough to provide three parameter-sharing schemes. our adversarial evaluation in the experiments is inspired by the work @xcite in which the authors proposed an adversarial approach to measure the utility of the image in multi-modal translation. in this work, a random image is fed into the model instead of the paired one. then the difference in performance reflects the importance of visual information. we apply a similar adversarial evaluation to our models by randomizing the visual object images in the training stage. to fully exploit visual modality, we degrade the linguistic context and reconstruct the original text from both modalities. this strategy leads to a similar model framework to the vision-language pretrained models @xcite @xcite . however, vision-language pretrained models can only initialize the encoder while our methods provide various models to learn cross-modal representations in the encoder-decoder architecture. ## conclusion in this paper, we have proposed an entity-level cross-modal learning approach that explicitly incorporates visual information into linguistic entities and is combined with the text-only translation task in a multi-task framework. our extensive results show that our models can achieve comparable or even better performance than state-of-the-art models. furthermore, we take an in-depth analysis to figure out why the approach works by contrasting the translation correctness of entity words with multiple adversarial models and former mmt models. the results show that the translation accuracy of entity words significantly increases with the help of entity-level visual information. our findings suggest that images can be utilized explicitly in an mmt model and better approaches are favored to leverage the fine-grained object information in the image.