Make the Blind Translator See The World: A Novel Transfer Learning Solution for Multimodal Machine Translation (2021)

Abstract: Based on large-scale pretrained networks and the liability to be easily overfitting with limited labelled training data of multimodal translation (MMT) is a critical issue in MMT. To this end and we propose a transfer learning solution. Specifically and 1) A vanilla Transformer is pre-trained on massive bilingual text-only corpus to obtain prior knowledge; 2) A multimodal Transformer named VLTransformer is proposed with several components incorporated visual contexts; and 3) The parameters of VLTransformer are initialized with the pre-trained vanilla Transformer and then being fine-tuned on MMT tasks with a newly proposed method named cross-modal masking which forces the model to learn from both modalities. We evaluated on the Multi30k en-de and en-fr dataset and improving up to 8% BLEU score compared with the SOTA performance. The experimental result demonstrates that performing transfer learning with monomodal pre-trained NMT model on multimodal NMT tasks can obtain considerable boosts.

## introduction transformer-based models using large-scale parallel corpora have significantly improved the performance of neural machine translation (nmt), marking an important milestone @xcite . additionally, multimodal machine translation (mmt) incorporating image signals into rnn-based encoder-decoder shows improvements on translation quality due to the forceful disambiguation @xcite . in this paper, we aim to investigate, on top of transformer, whether the paradigm of first pretraining and then fine-tuning can be effectively applied to mmt, concretely transferring from monomodal to multimodal tasks. constant attention has been paid on mmt task @xcite in the conference of machine translation (wmt) in recent years @xcite @xcite . formally, it aims to learn a function mapping: x × @xmath0, which takes source text and an image as input and translate them into the target text as shown in figure 1 . additional modality is to disambiguate the source sentence, with the reference of image. however, the effectiveness of the visual context has been questioned by prior work @xcite @xcite . they show that visual context is not convincingly useful and the marginal gain is pretty modest, which is speculated to be resulted from the limitation of available datasets -the large-scale text-only bilingual corpus limited multimodal bilingual corpus pre-training fine-tuning + vision text + image figure 2: the multimodal transfer learning solution. 1) initialize a vanilla transformer. 2) train the model with large-scale parallel corpus. 3) add visual related components. 4) fine-tune the model on limited multimodal corpus. for the text-only nmt tasks, the transformer @xcite provides a novel architecture on language generation which supersedes rnn architectures rapidly with enhanced parallelizability. meanwhile, the framework of pre-training and fine-tuning becomes a standard pipeline since bert @xcite achieved the sota performances over a bunch of natural language understanding tasks. this to some extent suggests that transfer learning could effectively solve nlp tasks which requires deep understanding on the semantics but have limited size of in-domain data. therefore, in this paper, we will investigate whether it's feasible to apply transfer learning to mmt task, i.e. transferring the prior knowledge learned from monomodal task into a multimodal task, as shown in figure 2 . the contribution of our work can be summarized as follows: ## related work there are a spectrum of prior works investigating mmt. @xcite used standard rnn encoder-decoder with attention @xcite to fuse textual and visual features. both of them employed pretrained image classification models like vgg and resnet to extract visual features and combine with textual features with different schemes of attentions. imaginet is proposed to predict the visual feature conditioned on textual inputs, which is used to improve the quality of the representation of contexts @xcite , where they decompose the mmt task into two sub-tasks where each can be trained separately with large external corpus. @xcite extends the work of imagination by converting the decoding process into a similarity based searching between the predicted embedding and the embedding of the vocabulary, which is achieved by optimizing a marginal loss on pre-trained word embeddings with predicted word embeddings. besides, @xcite @xcite ) make comprehensive summaries on the mmt tasks from mmt 2016 to 2018, which shows two major findings from the task: 1). the effectiveness of the additional modality is still questionable or limited, which encourages researchers to go further on the usage of visual information. 2). fine-grained evaluation metrics have to be adopted to evaluate the true impact of the multimodality. there are still some impressive works built upon transformer-based architecture. memad @xcite achieves the best performance on flickr16 and flickr17 test sets with a multimodal transformer model, which is pre-trained on massive out of domain data including opensubtitles and ms-coco captions. they perform comprehensive experiments on the model with different data and model settings. @xcite proposes the method named universal visual retrieval which builds a look up table from topic word and image with tf-idf. before translation, m images are retrieved from the image set. then, visual features will be aggregated with textual features to produce the hidden states. the umnmt proposed in @xcite makes it possible to train a mmt model with bilingual but non-paired corpus and images. in their work, each language has an encoder and a decoder but shares one image encoder. they use the cycle-consistency loss to train the model by translating the text into target language, then, recover it back. in summary, many approaches are proposed to tackle the mmt task from following two direction: however, we find that the pre-training and fine-tuning framework is under-investigated for mmt tasks, especially the cross-modal pre-training, which motivates us to explore in this work. ## vltransformer first of all, we briefly review the architecture of transformer @xcite . in the transformer, source texts are fed into the encoder and transformed into vectors with the word embedding and positional embedding, then, n layers of multi-head attention blocks are applied encoder decoder 0 0 the man . . . <@xmath1... ... ... to produce the hidden states h. for the decoder, the previously generated tokens until step t will be fed into the decoder to interact with the context h to predict the token of step t + 1. more formally, the encoding and decoding process is denoted as follows: ## analysis we compare our results with another six latest methods in although the attention map looks good, we actually manually amplify the score of visual features, in the experiment, we find that the model is more inclined to get contextual information from text instead of image although we have already used cross-modal masking. some reasons can be speculated: 1) the size of training data is relatively small which means the newly initialized visual related parameters can not be fully trained. 2) we investigate the extracted detected objects and find out that there are mistakes in the detection which actually leads noise into the model. ## conclusion we propose a cross-modal transfer learning solution to take full advantage of pre-trained monomodal model in the multimodal task. the approach of cmm to incorporate visual information into translation achieves remarkable results in the mmt tasks evaluated on multi30k dataset, which reveals that prior knowledge of monomodal data can be transferred in a multimodal model even if fine-tuning on limited multimodal data. furthermore, the shared encoder demonstrates perfect compatibility with the newly introduced visual features, which encourages us to dig into methods for visual and textual alignment with transformer architectures. to sum, we show the evidence that our model is able to decode from both modalities after fine-tuning with the cross-modal masking method.