RC 3: Regularized Contrastive Cross-lingual Cross-modal Pre-training (2023)

Abstract: Multilingual vision-language (V&L) pre-training has achieved remarkable progress in learning universal representations across different modalities and languages. In spite of recent success, there still remain challenges limiting further improvements of V&L pre-trained models in multilingual settings. Particularly, current V&L pre-training methods rely heavily on strictly-aligned multilingual image-text pairs generated from English-centric datasets through machine translation. However, the cost of collecting and translating such strictly-aligned datasets is usually unbearable. In this paper, we propose Regularized Contrastive Cross-lingual Cross-modal (RC3) pre-training, which further exploits more abundant weakly-aligned multilingual image-text pairs. Specifically, we design a regularized cross-lingual visio-textual contrastive learning objective that constrains the representation proximity of weakly-aligned visio-textual inputs according to textual relevance. Besides, existing V&L pre-training approaches mainly deal with visual inputs by either region-of-interest (ROI) features or patch embeddings. We flexibly integrate the two forms of visual features into our model for pre-training and downstream multi-modal tasks. Extensive experiments on 5 downstream multi-modal tasks across 6 languages demonstrate the effectiveness of our proposed method over competitive contrast models with strong zero-shot capability.

## introduction vision-language (v&l) pre-training aims to learn universal representations that can express visual and textual semantics informatively. it exploits a large amount of multi-modal data (e.g. imagetext pairs) to make the model capable of handling cross-modal data. till now, the advents of various v&l pre-trained models have achieved remarkable results on many downstream multi-modal tasks. recently, v&l pre-trained models have developed from focusing on english-dominant tasks @xcite @xcite into multilingual scenarios @xcite @xcite . to this end, researchers construct multi-modal data in multiple languages and design various cross-lingual pre-training objectives. such advances enable multi-modal modelling to leverage more diverse language resources. meanwhile, these multilingual v&l pre-trained models also show their advantages over previous english-centric models in terms of generalization abilities across languages, especially in zero-shot settings. despite the promising performances of current multilingual v&l models, one of the major challenges is that they usually require massive strictlyaligned multilingual image-text pairs. the prevalent practice is to translate english-only multimodal datasets into pseudo-parallel multilingual versions via machine translation (mt) @xcite . however, the cost of collecting and translating such large-scale multimodal datasets is often unbearable. to deal with this issue, we turn our eyes on those more easily available weakly-aligned multilingual multi-modal data, such as wit @xcite . as shown in figure 1 , the so-called "weakly-aligned" means that the multilingual textual data of the same image are not strictly parallel. in this paper, we propose a regularized contrastive cross-lingual cross-modal (rc 3 ) pretraining framework, which can make better use of relatively abundant weakly-aligned multilingual image-text pairs. specifically, we adopt an encoder-decoder architecture so that our model can be more adaptive to both discriminative and generative downstream tasks. besides the widely used image-text matching (itm) task, we further introduce masked conditional language modelling (mclm) and cross-lingual textual contrastive learning (xtcl) along with our proposed regularized cross-lingual visio-textual contrastive learning (r-xvtcl) during pre-training. particularly, while r-xvtcl encourages the visio-textual representations of two weakly-aligned image-text pairs to be close, a regularization term is designed to constrain such proximity according to the textual relevance of their respective texts. meanwhile, in current v&l models, there are mainly two ways of processing visual inputs:(1) region-of-interest based (roi-based). it uses external object detectors (e.g. faster-rcnn @xcite ) to extract roi features from images and feed them with paired text into v&l models @xcite @xcite @xcite . this method exerts the informativeness of roi features, but such cumbersome protocol hinders the usage of massive online image-text pairs and requires additional procedures for various downstream tasks. our contributions can be summarized as follows: ## our approach in this section, we first briefly introduce the three types of datasets used for pre-training and more details are given in appendix a. then, we describe the model architecture and pre-training objectives. ## ablation study in this section, we conduct ablation studies to investigate the effect of our proposed training objectives in section 2.3. adopting roi-based visual features, we investigate the following three model variants: from in recent years, there have been a series of v&l pre-trained models achieving remarkable progress on many downstream multi-modal tasks. overall, these studies adjust model architectures and design various pre-training objectives to learn alignment between visual and textual modalities. they can be mainly classified into single-stream @xcite @xcite and twostream v&l architectures @xcite . apart from the above models, some multilingual v&l pre-trained models are proposed to learn universal representations across multiple languages and modalities. one of the major difficulties is the lack of high-quality multilingual multi-modal pretraining data. to address this issue, @xcite proposed to integrate multilingual pre-training and multi-modal pre-training. concretely, batches of multilingual text corpora and monolingual multimodal data are alternately used. following a similar manner, @xcite build muniter and xuniter by initializing model parameters with mbert and xlm-r, respectively. furthermore, @xcite translate original english pretraining data into multiple languages and propose uc 2 to learn universal representations by introducing two cross-lingual/cross-modal pre-training tasks. these models leverage strictly-aligned multilingual and multi-modal datasets that are relatively difficult to collect. therefore in this paper, we additionally make better use of more abundant weaklyaligned multilingual multi-modal data. ## conclusion in this paper, we propose regularized contrastive cross-lingual cross-modal pre-training, which additionally exploits relatively more abundant weakly-aligned multilingual image-text pairs. during pre-training, we constrain the proximity of visio-textual representations of weakly-aligned image-text pairs according to their textual relevance. besides, we further enhance our v&l model by integrating roi-based and patch-based visual features. compared with recent competitive v&l models, our model achieves higher or comparable results, especially demonstrating stronger zero-shot performance.