Efficient Multilingual Multi-modal Pre-training through Triple Contrastive Loss (2022)

Abstract: Learning visual and textual representations in the shared space from web-scale image-text pairs improves the performance of diverse vision-and-language tasks, as well as modality-specific tasks. Many attempts in this framework have been made to connect English-only texts and images, and only a few works have been proposed to extend this framework in multilingual settings with the help of many translation pairs. In this multilingual approach, a typical setup is to use pairs of (image and English-text) and translation pairs. The major limitation of this approach is that the learning signal of aligning visual representation with under-resourced language representation is not strong, achieving a sub-optimal performance of vision-and-language tasks. In this work, we propose a simple yet effective enhancement scheme for previous multilingual multi-modal representation methods by using a limited number of pairs of images and non-English texts. In specific, our scheme fine-tunes a pre-trained multilingual model by minimizing a triplet contrastive loss on triplets of image and two different language texts with the same meaning, improving the connection between images and non-English texts. Experiments confirm that our enhancement strategy achieves performance gains in image-text retrieval, zero-shot image classification, and sentence embedding tasks.

## introduction transferring visual representations learned from a large annotated set into downstream tasks of interest @xcite is the standard approach to achieve the state-of-the-art performance @xcite . however, due to the labeling cost, the scalability of this approach is rather limited. in contrast, self-supervised learning with contrastive losses @xcite has proven to learn semantic representations without using the explicit labels, which becomes a promis-ing solution to obtaining general visual representations. in addition, this paradigm combined with billion-scale unlabeled samples is competitive with the annotation-based transfer learning approaches in multiple tasks @xcite . in natural language processing (nlp), pretraining with billions of corpus and transferring to downstream tasks has likewise achieved tremendous success. and it has been a de facto standard for a recent decade @xcite @xcite . witnessing successes in two domains, researchers have been actively attempting to find visual-and-language (vl) representations by combining supervisions that come from billions of both images and languages. @xcite @xcite have attempted to utilize highly curated vl datasets such as ms-coco @xcite or visual genome @xcite . however, the non-trivial collection process is the major limitation in scaling up these datasets. clip @xcite overcame the limitation by learning vl representations from web-scale 400m image-totext pairs on both visual and vl downstream tasks, even with a simple contrastive loss. align @xcite extended this approach by scaling up image-text pairs to 1.8b with simple filtering compared to clip and showed state-of-the-art scores on both visual and visual-language tasks. such web-scale approaches as well as the earliest attempts have focused on connecting images to english texts. using a separate translator might be a practical solution to match images and multilingual texts, but this resorts to sub-optimal results because the effect of translation errors is not explicitly considered in the vl representation learning process, and translating every text into dozens of languages is inefficient. in language models, many attempts have been successfully made to develop language-agnostic representations, improving the performance of downstream tasks on under-resourced languages. in order to achieve this goal, @xcite ; @xcite ; @xcite pre-train language models over a multilingual corpus for transferring knowledge across languages. @xcite ; @xcite utilize bilingual translation pairs to transfer the information from common languages to under-resourced ones in a more efficient way. following this research direction in language models, mural @xcite suggests the efficient multilingual vl modeling that leverages both the alignments of (a) monolingual image-totext and (b) multilingual text-to-text on the training dataset simultaneously. however, since english is the only language that has a direct connection with images, the performance of non-english languages on several multi-modal tasks is relatively weaker than that of english. linking all languages directly to images would be the easiest and ideal solution, but obtaining billions of image-text datasets for all languages is nearly impossible, especially for low-resourced languages. to tackle the data-scarcity problem, we propose a simple but efficient enhancement via triplet contrastive learning (etcl) that utilizes multi-modal zero-shot transfer through relatively small amounts of image-text datasets in non-english languages through a triplet contrastive loss. in etcl, we further train the pre-trained multilingual vl model with a new limited number of pairs of image and non-english texts to strengthen the weak alignment of the pre-trained model. in order to fully leverage multi-modal cross-lingual zero-shot transfer, we introduce a triplet contrastive learning which considers (a) image-texta, (b) texta-textb and (c) textb-image, where texta and textb are multilingual translation pairs. through various experiments, we show that our framework provides significant performance improvement for various languages in multi-modal retrieval tasks and zero-shot image classification and sentence embedding tasks. interestingly, the large performances gains also occur in some languages which are not included in the training dataset for etcl. through ablation study, we prove that the phenomenon comes from the multi-modal crosslingual zero-shot transfer with regard to grammatical and geographic relationships between languages. and we show that etcl leverages the relationships efficiently. in summary, our contributions to the multilingual vl representation are: ## related works we briefly review the research areas and key references most relevant to our approach. ## approach this section describes the details of conventional multilingual vl modeling, the proposed etcl scheme, and training details. text encoder text encoder shared der welpe schläft auf der bettdecke. 小狗睡在羽绒被上。 triplet contrastive loss ## experiments in this section, we present experiment settings to evaluate our proposed model over vl, visual, and language tasks. ## results as presented in our contributions in section 1, our study has different goals: (1) investigate the performance gains based on the proposed method on various downstream tasks including image-text retrieval, zero-shot image classification and sentence retrieval (2) study the effect of etcl in related languages (geographically and grammatically), ( 3 ) check whether etcl activates information sharing in multilingual training. by taking into account the proposed goals, in this section, we report our experimental results in various tasks. ## conclusion vl modeling using large-scale web-crawled datasets has shown great success but cannot be easily utilized for low-resourced languages due to the lack of data. although a method to efficiently create a multilingual vl model through indirect text-text alignment has been proposed, the vl representations of low-resourced languages are still weaker than that of english. this work proposes a new approach to solve the limitation based on the observation that multi-modal zero-shot transfer occurs with regard to grammatical and geographical relationships between languages. our proposed method can be easily adapted to multilingual multimodal models trained similarly to mural.