Cross-lingual Cross-modal Pretraining for Multimodal Retrieval (2021)

Abstract: Recent pretrained vision-language models have achieved impressive performance on cross-modal retrieval tasks in English. Their success, however, heavily depends on the availability of many annotated image-caption datasets for pretraining, where the texts are not necessarily in English. Although we can utilize machine translation (MT) tools to translate non-English text to English, the performance still largely relies on MT’s quality and may suffer from high latency problems in real-world applications. This paper proposes a new approach to learn cross-lingual cross-modal representations for matching images and their relevant captions in multiple languages. We seamlessly combine cross-lingual pretraining objectives and cross-modal pretraining objectives in a unified framework to learn image and text in a joint embedding space from available English image-caption data, monolingual and parallel corpus. We show that our approach achieves SOTA performance in retrieval tasks on two multimodal multilingual image caption benchmarks: Multi30k with German captions and MSCOCO with Japanese captions.

## introduction recent pretrained vision-language models @xcite @xcite @xcite based on transformer @xcite have achieved remarkable performance on cross-modal retrieval @xcite @xcite , image captioning @xcite and visual question and answering (vqa) @xcite tasks in english. for instance, most leading competitors in the vqa contest foot_0 rely on the transformer-based pretrained vision-language models. however, their success heavily depends on the availability of a large amount of annotated imagecaption pretraining datasets (e.g., conceptual cap-tions @xcite ). in reality, there are limited such data in other languages. when generalizing to cross-lingual cross-modal downstream tasks, a straightforward way is to utilize machine translation (mt) tools to translate non-english text to english and reuse the fine-tuned models in english. nevertheless, the performance strongly relies on the mt tool's capability and suffers from high latency problems in real-world applications. to learn multilingual multimodal representations, recent researchers utilized multilingual datasets to model images and text captions in a joint embedding space. based on how the shared feature space is learned, there are two categories: wordlevel alignments @xcite and sentence-level alignments @xcite . those models can capture a certain level of semantic similarity among languages and images. they, however, only modeled the relevance of text and images in a global manner. such a limitation may prevent these models from effectively detecting relevance locally. in parallel, cross-lingual language models such as multilingual bert @xcite and xlm @xcite , and pretrained vision-language models @xcite @xcite have been prevalent in bridging different languages and modalities. those models use the transformer @xcite architecture simultaneously pretrained from multiple languages or image-caption pairs to construct an encoder, and then fine-tune the encoder on downstream applications with task-specific objectives. the whole process enables sufficient interaction across languages and other modalities via crossattention. however, current cross-lingual models and cross-modal models are trained separately on multilingual corpus and english-caption data. hence the resulting pretrained models are not directly applicable to downstream cross-modal tasks involving non-english languages. ## methodology our framework adopts the network structure of vl-bert @xcite . vl-bert is a singlestream cross-modal model that concatenates word features from the text and bounding box features from the image and feeds the concatenated sequence into a series of transformer blocks. ## experiment for pretraining, we utilize two public english image-caption datasets: sbu captions @xcite and conceptual captions @xcite . due to broken urls, we only collected around 3.7m text-image pairs in total. for monolingual (en, de, ja) text and parallel corpus (en-de), we randomly sample 20m sentences from wikipedia text foot_1 and 9m parallel sentences from multiun corpus foot_2 . we also collected 2.8m en-ja parallel sentences from @xcite . for fine-tuning, we use two multilingual multimodal benchmarks for retrieval, mscoco (en, ja) @xcite and multi30k (en, de) @xcite . mscoco contains 123, 287 images, and each image contains five captions. following the settings in @xcite , we split the english data into 113, 287 training samples, 5, 000 validation samples, and 5, 000 testing samples. @xcite generated the japanese captions for a subset of 33, 745 images. similarly, we split 23, 745 samples for training, 5, 000 for validation as 5, 000 for testing. multi30k contains 31, 783 images, with each having five captions as well. following @xcite , we split the dataset into 29, 783 training samples, 1, 000 validation samples and 1, 000 testing samples. we use r@k (@xmath0,5,10) as evaluation metrics. r@k is the percentage of ground-truth matchings appearing in the top k-ranked results. ## conclusion in this work, we introduce multilingual corpus and three pretraining objectives to improve transformer based vision-language models for retrieval tasks. extensive experiments demonstrate the effectiveness of our contributions on cross-modal retrieval tasks. detailed ablation studies justify our modeling choices. our future work is to explore the zero-shot transferring capability of our framework.