Adding multimodal capabilities to a text-only translation model (2024)

Abstract: While most current work in multimodal machine translation (MMT) uses the Multi30k dataset for training and evaluation, we find that the resulting models overfit to the Multi30k dataset to an extreme degree. Consequently, these models perform very badly when evaluated against typical text-only testing sets such as the newstest datasets. In order to perform well on both Multi30k and typical text-only datasets, we use a performant text-only machine translation (MT) model as the starting point of our MMT model. We add vision-text adapter layers connected via gating mechanisms to the MT model, and incrementally transform the MT model into an MMT model by 1) pre-training using vision-based masking of the source text and 2) fine-tuning on Multi30k. We achieve a state-of-the-art performance on the Multi30k 2016 en-de test set of 46.5 BLEU4 score and 0.61 CoMMuTE score via this approach while retaining the performance of the original text-only MT model against the newstest dataset.

## introduction the task of multimodal machine translation (mmt) is to automatically translate text while using additional modalities (e.g., image, video, audio) to aid in translation. prior work has shown that mmt can use contextually relevant images to aid in translation of sentences that contain ambiguities or missing textual information @xcite . for example, the noun "bank" is ambiguous and contextually dependent in english ("financial institution" or "river edge") but unambiguous in french ("banque" or "rive"). the hypothesis that these ambiguities or missing information can be resolved with contextually relevant images is persuasive. much work in mmt @xcite @xcite focus on the multi30k dataset @xcite , a dataset comprising 30,014 image captions and corresponding translations in different languages. however, compared to the domain of text-only translation where mt models are trained using millions of examples, the multi30k dataset is an extremely small dataset. consequently, the mmt models will naturally overfit to the multi30k dataset and perform poorly against testing sets that text-only translation models are typically evaluated against (section 4). text-only machine translation is a much larger domain than multimodal machine translation and many strong models have been developed in the field @xcite . thus, using a pre-trained textonly model as a starting point for mmt is a promis-ing approach to advance the state of mmt. to demonstrate this, we incrementally transform a text-only mt model into an mmt model, resulting in stateof-the-art performance against the multi30k dataset while retaining the performance of the pre-trained model against text-only test sets. we use a pre-trained transformer-based translation model as our starting point. we evolve this text-only translation model into an mmt model using adapters @xcite and gating mechanisms such that the model learns how to use visual information while preserving its original translation performance. we do this by 1) combining a strong pre-trained translation model and a pre-trained vision-language model to create an mmt model, 2) pre-training the mmt model on a dataset of captions augmented with informed visual grounding and machine generated translations along with a dataset collated from a text-only mt dataset, and 3) finetuning against the multi30k dataset. using this model architecture and training process, we achieve high performance against the multi30k test sets while retaining high performance against text-only testing sets ( ## methods we take a similar approach that @xcite used to create their generative vision-language model, flamingo, while adapting their approach for the mmt task. flamingo is a generative decoder-only visionlanguage model created by combining a pre-trained generative language model and a pre-trained vision model, where vision and text interactions are mod-eled by via gated vision-text cross-attention layers inserted before each decoder layer. then, the model is incrementally converted from using only text information to using both vision and text information by freezing the pre-trained portions of the model. the gating values are set to 0.0 at the beginning of training in order that the vision-language model initially performs equivalently to the language model, and as training progresses the gating values diverge from 0 via back-propagation and consequence learns to use vision information gradually. vision-text layer encoder layer perceiver resampler layer figure 1 : multimodal translation architecture, where multimodal components are incorporated into the transformer translation model introduced by @xcite . the parameters in the model bordered by red are initialized randomly and updated for training, while the parameters in the pre-trained vision encoder and the pre-trained transformer translation model bordered by black are frozen. the gating parameters in the vision-text layers are updated using back-propagation, allowing us to smoothly transition from a text-only translation model into a multimodal translation model. analogously, we start from a pre-trained transformer-based text-only mt model and a pretrained vision model to create an mmt model by inserting a vision-text cross-attention layer before each encoder layer. using trainable gating parame-ters, we incrementally convert the model from using only text information to using both vision and text information to perform translation. we call our model gram @xcite . while trainable gating parameters have been used in previous work for mmt @xcite @xcite , our work is unique in the much lower number of gating parameters and in that it allows for the smooth transition of the model from performing as an mt model to performing as an mmt model. both the flamingo model and our model were trained using the next-token prediction task, as is typical for text-only machine translation. unlike flamingo, which is a decoder-only model, our model is an encoder-decoder model. we inserted the visiontext layers before each of the encoder layers only, as we found it to perform better than inserting visiontext layers before the decoder layers only or before both the encoder and decoder layers (appendix d.1). aside from the perceiver resampler module and the gated vision-text cross attention layers used in flamingo model, which we use to convert our model from an mt model to an mmt model, our gram model follows the original text-only transformer mt model's hyper-parameters, layers, and training objectives as closely as possible. ## results and discussion we use the evaluation framework proposed by @xcite , where they argued that mmt models should be evaluated by measuring both 1) their use of visual information to aid in the translation task and 2) their ability to translate complex sentences as is done for text-only machine translation. we evaluate model performances against 1) the commute @xcite test set, 2) the multi30k @xcite test sets, and 3) the wmt news translation task @xcite test sets (newstest) using commute score and bleu4 calculated using sacrebleu @xcite . the main evaluation results are shown in m cr is our model pre-trained on the cr dataset; m cr,m30k is our model pre-trained on cr and finetuned on multi30k; m m30k is our model trained on multi30k without the pre-training step; gated fusion and rmmt are our evaluations of the models published by @xcite ; vgamt is the reported performance of the model published by @xcite . "text inputs only" shows performance of when only the source text is given. "multimodal inputs" shows the performances when both source text and image is used as input. "non-matching inputs" shows performance when source text along with a random image is used as input. ## conclusion text-only machine translation is a much larger domain than multimodal machine translation and many strong models have been developed in the field. the approach of transforming a language model into a vision-language model was successful demonstrated via flamingo, and thus have a high probability of working well in the similar task of machine translation. following this idea, we designed an mmt model that began as a performant text-only mt model and incrementally transformed it into a mmt model by 1) pre-training using informed vision-based masking of the source text and 2) finetuning on multi30k. we achieved a state-of-the-art performance on the multi30k 2016 test set of 46.5 bleu4 score via this approach while retaining high performance against commute and the newstest datasets. there are many approaches for improving our model including the training process, where the pre-training dataset can be improved using more textonly datasets or augmenting text-only datasets using image retrieval, and model architecture, where techniques such as vmlm can be used to further enforce the use of image information in the model.