Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training (2023)

Abstract: Multilingual Vision-Language Pre-training (VLP) is a promising but challenging topic due to the lack of large-scale multilingual image-text pairs. Existing works address the problem by translating English data into other languages, which is intuitive and the generated data is usually limited in form and scale. In this paper, we explore a more practical and scalable setting: weakly supervised multilingual VLP with only English image-text pairs and multilingual text corpora. We argue that the universal multilingual representation learned from texts allows the cross-modal interaction learned in English to be transferable to other languages. To this end, we propose a framework to effectively unify cross-lingual and cross-modal pre-training. For unified modeling on different data, we design an architecture with flexible modules to learn different interactions. Moreover, two unified tasks are introduced to efficiently guide the unified cross-lingual cross-modal learning. Extensive experiments demonstrate that our pre-trained model learns universal multilingual multimodal representations, allowing effective cross-lingual transfer on multimodal tasks. Code and models are available at https://github.com/FudanDISC/weakly-supervised-mVLP .

## introduction in recent years, self-supervised pre-training technology has been studied extensively in various fields. the pre-trained models are able to encode generalized contextual representations for texts @xcite @xcite , images @xcite , and image-text pairs @xcite @xcite , further facilitating the downstream tasks and research. however, most pretraining studies are limited to english corpora. in order to overcome the language barrier and benefit a wider range of audience, it is important to extend the success of english-centric research to a multilingual scenario. recent works have demonstrated the effectiveness of cross-lingual language modeling @xcite . based on large-scale multilingual corpora, the models are able to learn universal representations for texts in multiple languages. however, large-scale and tightly associated multilingual image-text pairs are unavailable and costly to acquire. therefore, it is not straightforward to transfer existing vlp methods to other languages. as shown in figure 1 (a, b), previous works @xcite address the problem by transferring english data to other languages through different data augmentation strategies (e.g., code switch or translation engines), and then perform vlp on the generated multilingual data. despite being simple and intuitive, these methods have limitations since the augmentation is either constrained to specific forms that differ from natural data, or it is time-consuming to ensure the equality, making it difficult to scale effectively to more languages and larger datasets. meanwhile, large volumes of natural language texts are readily accessible in various languages as shown in figure 1 @xcite . this raises a question: can we relax the requirement for multilingual image-text pairs and use existing multilingual text resources to transfer english vlp to other languages? in this paper, we explore this weakly supervised setting to develop a more scalable multilingual vlp framework. we argue that by unifying crosslingual text modeling in cross-modal models, universal representations can be learned for multiple languages, and thus the cross-modal modeling ability learned from english image-text pairs can be easily transferred to other languages. the biggest challenge for such unification is that data in different forms (i.e., different languages or modalities) have different intrinsic properties. therefore, the key issue is how to effectively incorporate multilingual text pre-training into the vlp framework without conflicts among different data streams. from the model perspective, if we simply feed all data to a vanilla transformer model @xcite , different desired interactions are entangled in self-attention layers and may compete with each other during pre-training. to address this issue, we propose to disentangle different functionalities into different modules. specifically, we design a novel architecture by incorporating pluggable cross-attention layers into standard transformer layers. these layers can be activated to perform cross-modal and cross-sentence modeling or skipped for unpaired text modeling. multilingual text learning can thus transfer the self-attention to fit more languages, indirectly requiring the crossattention to adapt to the universal representations rather than competing with cross-lingual modeling in self-attention. in terms of training, vlp and language modeling methods tend to optimize different objectives. for unified pre-training, we introduce two tasks that share unified formulations for different data streams and guide unified cross-lingual and crossmodal learning. before the cross-modal fusion, we propose unified contrastive learning to simultaneously align parallel sentences in different languages and english image-text pairs, making it easier for the upper encoder to learn interactions shared across languages and modalities. on top of the whole model, we consider three self-supervised tasks: cross-lingual masked language modeling on unpaired texts to achieve universal multilingual representations, visual language modeling on english image-text pairs to learn cross-modal interaction, and translation language modeling on parallel sentences to enhance cross-lingual alignment. three tasks are further unified as a mask modeling task and cross-lingual learning is naturally unified with cross-modal learning. our contributions can be summarized as follows: ## conclusion in this paper, we explored weakly supervised multilingual vlp without multilingual image-text pairs. we proposed a flexible architecture and unified tasks to effectively unify cross-lingual modeling on multilingual texts and cross-modal modeling on english image-text pairs. experimental results validate the effectiveness of our approach to learn universal multilingual multimodal representations.