Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions (2021)

Abstract: Pre-trained contextual vision-and-language (V&L) models have achieved impressive performance on various benchmarks. However, existing models require a large amount of parallel image-caption data for pre-training. Such data are costly to collect and require cumbersome curation. Inspired by unsupervised machine translation, we investigate if a strong V&L representation model can be learned through unsupervised pre-training without image-caption corpora. In particular, we propose to conduct “mask-and-predict” pre-training on text-only and image-only corpora and introduce the object tags detected by an object recognition model as anchor points to bridge two modalities. We find that such a simple approach achieves performance close to a model pre-trained with aligned data, on four English V&L benchmarks. Our work challenges the widely held notion that aligned data is necessary for V&L pre-training, while significantly reducing the amount of supervision needed for V&L models.

## introduction pre-trained contextual vision-and-language (v&l) models @xcite @xcite @xcite have achieved high performance on various v&l tasks. however, different from contextual language models, such as bert @xcite , which are trained on easily-accessible unannotated text corpora, existing v&l models are still a step away from self-supervision. they require a massive amount of aligned text-image pairs for "mask-and-predict" pre-training. such aligned data are costly to collect and hard to scale up. for example, the widely used ms-coco dataset @xcite requires extensive * the two authors contributed equally. we are inspired by works on multi-lingual contextual language models @xcite . if we treat an image as a set of regions and each region as a visual token @xcite , v&l models share a similar goal with multi-lingual models as they both learn shared representations across different domains. although a multi-lingual language model pre-trained on non-parallel corpora such as mbert @xcite cannot align or translate languages out-of-the-box, its representation spaces for different languages can be easily aligned with a linear probe @xcite . this property suggests the existence of universal latent symmetries in the unaligned contextual embedding spaces and is believed to contribute to given text, the model is trained to predict masked words; given an image, the model is trained to predict masked regions and detector tags. the semantic class "cake" appears in both the language modality and the visual modality and is linked through the detector tags. note that we do not require a text segment with the word cake to appear together with the image. rather, we assume that as long as the text corpora are general enough, the word cake will appear in the textual modality eventually. the model can thus learn v&l representations from such weak supervision signals. we propose unsupervised v&l pre-training with unaligned text and images (see an illustration in figure 1 ). specifically, we take visualbert @xcite as a running example and apply unsupervised pre-training, resulting in unsupervised visualbert (u-visualbert). the model takes the form of a single transformer that can accept inputs from both modalities. during each step of pre-training, unlike the existing models that observe a batch of text-image pairs, our model observes either a batch of text segments or a batch of images. when provided with text, part of the text is masked and the model is trained to predict the masked words; when provided with an image, part of the image regions are masked and the model is trained to predict properties of the masked regions. to further encourage cross-modal fusion, we leverage the tags from an object detector as "anchor points" @xcite . for every object, we append its detected tag as a word to the visual input. the mask-and-predict objective is applied to the tags. for instance, for the image in figure 1 , the model can observe "cake" appears naturally as a word, a tag, and an image region. the direct typing of image regions and words can be learned and serves as a starting point for further alignment. the function of the detector tags resembles that of the "overlapping vocabulary" in multi-lingual language models, i.e., identical strings that appear in differ-ent languages with the same meanings (e.g., "dna" appears in both english and french). as the "overlapping vocabulary" improves cross-lingual transfer @xcite , we argue the detector tags can improve cross-modal grounding. we first conduct controlled experiments by pretraining on an english image-caption corpus without providing the alignment, following unsupervised machine translation and image captioning @xcite . results on four english v&l benchmarks (vqa @xcite , nlvr 2 @xcite , flickr30k image retrieval @xcite , and refcoco+ @xcite ) show that u-visualbert achieves comparable performance as models with access to textimage pairs (section 4). additionally, our approach is effective in practical settings, 1) when using independently collected images and captions and 2) when using images and general-domain text (bookcorpus @xcite ) without any captions (section 5.1). quantitative and qualitative analysis confirms the anchoring effect of the detector tags (section 5.2). as a byproduct, we conduct preliminary experiments to show the promise of the approach in a semi-supervised setting, where a hybrid model pre-trained with both aligned and additional unaligned data surpasses a model pre-trained only on aligned data. (section 6). the above experiments demonstrate the wide applicability of our method. we will open-source the project under 2 related work pre-trained v&l transformers various v&l models that are pre-trained with a "mask-andpredict" objective on aligned text-image data have been proposed @xcite @xcite @xcite @xcite @xcite . two kinds of designs have been proposed. two-stream models @xcite @xcite utilize separate transformers @xcite for each modality and a cross-modality module is adopted. single-stream models @xcite @xcite ) directly input the text and visual embeddings into one single transformer. they have been widely used by downstream tasks @xcite . probing tasks @xcite ) confirm that they capture useful v&l information after pre-training. two studies also try to incorporate "tag" information during pre-training. oscar @xcite adds detected tags as additional signals when pretraining with aligned data. we, however, do so for pre-training with unaligned data and show that the tags serve a more important role in unsupervised pre-training (section 5.2). vivo @xcite targets novel object captioning. they use manually annotated image-tag data for pre-training and image-caption data for fine-tuning. we do not use manually annotated data and the tags are noisily generated by a detector. ## approach we first take supervised visualbert (s-visualbert) as an example and illustrate how a typical v&l model is pre-trained with aligned data. then we introduce unsupervised v&l pre-training, and the resulting model unsupervised visualbert (u-visualbert). ## experiment as the domain and quality of data may affect the model performance, the conventional practice in unsupervised learning is to use aligned corpora without providing alignments, allowing for controlled comparison with a supervised model. for example, unsupervised machine translation creates unaligned corpora by splitting up parallel corpora @xcite while unsupervised image captioning @xcite create unaligned corpus by shuffling images and captions from mscoco @xcite . following prior work, we first conduct experiments by using conceptual captions (cc) @xcite as the source of images and text for both the supervised and unsupervised model. later in section 5.1, we show that our method is effective when the images and captions are collected independently and when no caption text is used. u-visualbert the model is pre-trained with shuffled captions and images. at each training step, we sample either a batch of images or a batch of text. following vl-bert @xcite , we find it beneficial to include bookcorpus @xcite , a general-domain text corpus, during pretraining. in sum, u-visualbert is trained on 3m images from cc, 3m captions from cc, and 2.5m text segments from bookcorpus 6 . ## analysis in this section, we analyze the effect of the text data and the role of the detector tags. ## semi-supervised pre-training unsupervised pre-training in itself has great practical and research value in many domains where aligned data is scarce. as a byproduct, we wonder if the approach could find its use in a semisupervised setting, where we pre-train a model with both aligned data and unaligned data. ## conclusion in this paper, we explore unsupervised pre-training with unaligned data. we conduct mask-and-predict pre-training on textual data and visual data and the detector tags are used as anchor points to bridge the two modalities. experiments show that unsupervised pre-training can achieve performance similar to supervised pre-training.