Weakly Supervised Vision-and-Language Pre-training with Relative Representations (2023)

Abstract: Weakly supervised vision-and-language pre-training (WVLP), which learns cross-modal representations with limited cross-modal supervision, has been shown to effectively reduce the data cost of pre-training while maintaining decent performance on downstream tasks. However, current WVLP methods use only local descriptions of images, i.e., object tags, as cross-modal anchors to construct weakly-aligned image-text pairs for pre-training. This affects the data quality and thus the effectiveness of pre-training. In this paper, we propose to directly take a small number of aligned image-text pairs as anchors, and represent each unaligned image and text by its similarities to these anchors, i.e., relative representations. We build a WVLP framework based on the relative representations, namely RELIT, which collects high-quality weakly-aligned image-text pairs from large-scale image-only and text-only data for pre-training through relative representation-based retrieval and generation. Experiments on four downstream tasks show that RELIT achieves new state-of-the-art results under the weakly supervised setting.

## introduction vision-and-language pre-training (vlp) @xcite @xcite @xcite has received increasing attention in recent years for its great success on various vision-and-language tasks, such as visual question answering @xcite , cross-modal retrieval @xcite , and image captioning @xcite . different from other foundation models @xcite such as bert @xcite and mae @xcite that only require single-modality data, vlp models rely on largescale aligned image-text datasets @xcite @xcite to bridge the gap between the two modalities, which requires either extensive manual annotations or heavy data cleaning processes @xcite . the natural difficulty of obtaining paired data hinders the scale of cross-modal datasets, while the success of unimodal pre-trained models implies the potential to exploit the unlabeled data for pre-training. therefore, besides collecting more paired data, it is a worthwhile direction to explore how to utilize lowcost unimodal data with limited cross-modal supervision, i.e., weakly supervised vision-and-language pre-training (wvlp). the core challenge of wvlp is to establish the connection between the two modalities without using a large number of aligned image-text pairs. existing works on wvlp @xcite @xcite usually address this by taking object tags as anchors as they are in the form of text and cover the information of the image at the same time. they use tags to collect weakly-aligned image-text pairs from unaligned unimodal data for pre-training and achieve competitive results compared to standard vlp models, demonstrating that tags can effectively bridge the gap between the two modalities. despite its success, using object tags as anchors suffers from two limitations. first, tags are merely local descriptions instead of a complete representation of the whole image and text. second, the vocabulary of tags only includes common concepts, making it difficult to represent images with complex semantics @xcite . these limitations could deteriorate the quality of the weakly-aligned data (and possibly pre-trained models) based on the object tags. therefore, to further improve the performance of wvlp, we need to reconsider the choice of the cross-modal anchors and find a better approach to measure the alignment between an image and a text. recently relative representation has been proven to be effective in representation learning @xcite and zero-shot image classification @xcite . the main idea is to represent a data point as its similarities to a set of selected data points (anchors). we argue that relative representations can be a good choice for wvlp because (1) they are built on the semantic similarities of well-trained neural network representations rather than on superficial human-designed features like tags and (2) they are modality-invariant by design because they reflect the intrinsic relationships between data points, which naturally enables communication between different modalities. in this paper, we propose relit, a novel relative representation-based wvlp framework. instead of object tags, we directly use a minuscule amount (compared to pre-training data) of available imagetext pairs as anchors, and create a common relative representation space with respect to the anchors for unaligned images and text. this allows us to estimate the semantic similarity of any image-text pair by calculating their distance in the relative representation space. in addition, we design two relative representation-based data collection methods that can retrieve or generate weakly-aligned image-text pairs from unaligned unimodal corpora. experimental results prove the effectiveness of relative representations in bridging the gap between image and text modalities. moreover, our work reveals a promising research direction to establish crossmodal alignments by finding and aligning invariant data structures in different modalities, which may inspire future works on multimodal pre-training. our main contributions are as follows: relative representations. the concept of relative representations is initially proposed by @xcite . they show that the relative representations obtained from the representation spaces of different models are similar, which enables comparison and alignment between latent embeddings of different learning models. @xcite explore relative representations in a multimodal scenario to align images and text for zero-shot image classification tasks. specifically, they use 1.6m image-text pairs to build the relative representation space, which is comparable to the size of the data used in pre-training. to the best of our knowledge, our work is the first that exploits relative representations for weakly supervised cross-modal pre-training. weakly supervised vision-and-language pretraining. li et al. (2021b) first explore wvlp with unaligned image and text corpora and use object tags directly as pseudo captions for images to bridge the vision and language modalities. zhou et al. (2022) use tags to retrieve weakly-aligned captions for each image and then apply multi-granular alignment tasks on this retrieved dataset. wang et al. (2022b) propose the cross-modal cutmix to replace some grounded words with regions that have the same tags, and construct a multimodal view of the text-only sentences for pre-training. chen et al. (2022) introduce an end-to-end framework with a referring expression matching task. different from all of these wvlp works that utilize tags as anchors to provide object-level cross-modal alignment signals, our work uses relative representations to capture the overall semantic similarity between each image and text and demonstrates its effectiveness in wvlp. data augmentation. data augmentation has been extensively employed in various computer vision @xcite and natural language processing tasks @xcite . in the area of vlp, @xcite augment the noisy web-crawled aligned data by filtering low quality image-text pairs and generating synthetic captions with an image captioner fine-tuned on clean image-text pairs. in this work, we adopt a similar filter-and-generate process in the construction of weakly-aligned data for wvlp, but our relative representation-based pseudo caption generator is fine-tuned on the text-only dataset. ## conclusion this paper introduces the idea of relative representations to weakly-supervised vision-and-language pre-training and demonstrates its effectiveness in bridging the gap between the two modalities. we propose a relative representation-based framework that can both retrieve and generate weakly-aligned image-text pairs for pre-training. experimental results show that our method outperforms all previous tag-based approaches under the weaklysupervised setting. we hope our work will motivate future work in multimodal pre-training.