End-to-End Unsupervised Vision-and-Language Pre-training with Referring Expression Matching (2022)

Abstract: Recently there has been an emerging interest in unsupervised vision-and-language pre-training (VLP) that learns multimodal representations without parallel image-caption data. These pioneering works significantly reduce the cost of VLP on data collection and achieve promising results compared to supervised VLP. However, existing unsupervised VLP methods take as input pre-extracted region-based visual features from external object detectors, which both limits flexibility and reduces computational efficiency. In this paper, we explore end-to-end unsupervised VLP with a vision encoder to directly encode images. The vision encoder is pre-trained on image-only data and jointly optimized during multimodal pre-training. To further enhance the learned cross-modal features, we propose a novel pre-training task that predicts which patches contain an object referred to in natural language from the encoded visual features. Extensive experiments on four vision-and-language tasks show that our approach outperforms previous unsupervised VLP methods and obtains new state-of-the-art results.

## introduction vision-and-language pre-training (vlp) @xcite @xcite @xcite @xcite has achieved great success on a wide range of vision-and-language tasks, e.g., visual question answering @xcite , image-text retrieval @xcite and text-to-image generation @xcite . the major challenge for vlp is how to bridge the gap between the representations of vision and language modalities, which is typically ad-dressed by training on large-scale parallel imagetext datasets @xcite @xcite with specially designed pre-training tasks. however, these datasets require either extensive human annotations or massive data cleaning efforts, making them difficult to collect, especially when compared to the large amount of unimodal data. to alleviate this problem, there has recently emerged some works exploring unsupervised vision-and-language pre-training (uvlp), where only non-parallel image and text data is leveraged @xcite . specifically, @xcite propose to use image region features and their detected object tags produced by an object detector as pesudo-parallel pairs to bridge the gap between the two modalities. @xcite further enrich the training data with retrieved text pieces based on object tags and pre-train their model with multi-granular alignment tasks. these works achieve competitive results compared to several supervised vlp models, demonstrating the potential of uvlp. however, current research on uvlp adopts a two-step training strategy that first extracts regionbased image features with an external object detector and then builds a multimodal model based on the region features. this is considered to have several limitations for vlp. first, region features may be sub-optimal for vlp because they are designed for object detection tasks rather than general cross-modal understanding and are fixed in the pre-training process @xcite . second, the process of extracting region features is time-consuming, which significantly reduces the inference efficiency @xcite . finally, this two-step training strategy hinders the use of vision pre-trained models (v-ptms) such as vit @xcite and swin transformer @xcite , which are not off-the-shelf object detectors but achieve promising performance on general vision tasks. therefore, how to perform uvlp in an end-to-end manner, i.e., using raw images instead of region features, is still a valuable open question. to explore the question, we propose an endto-end uvlp framework named e2e-uvlp. the framework consists of a vision encoder and a pretrained language model (plm), both of which are pre-trained on unimodal data and they are connected by a linear projection layer. taking image patches as input, our framework is capable of leveraging a wide range of v-ptms. without using object tags in inference, the computational cost introduced by external object detectors is eliminated. inspired by previous works @xcite @xcite , we derive a masked tag prediction (mtp) pre-training task which predicts the masked object tags given a raw image and the other object tags detected from it. combining mtp with the widely used masked language modeling (mlm) task @xcite , we successfully make e2e-uvlp achieve comparable or better results than existing uvlp methods, justifying end-to-end uvlp is feasible. although the mtp task is effective, further investigation reveals that the obtained model is less effective when dealing with complex attributes of objects, e.g., locating objects or determining the relationship between objects in an image. we argue it is due to two pitfalls of the mtp objective: (1) discrepancy between training and inference: an object is referred to by its tag and numerically encoded position in training, while referred to only in natural language in inference. similar discrepancies have been shown to hurt performance significantly in plm studies @xcite . in summary, our contributions are three-fold: ## conclusion we propose a novel framework that performs end-to-end unsupervised vision-and-language pretraining without using costly and sub-optimal region features. to reduce the training-inference discrepancy, we propose a new pre-training task that predicts the locations of objects with synthetic referring expressions that are more similar to real text. experiments show that our approach consistently outperforms existing unsupervised visionand-language pre-training methods, and achieves competitive results compared to supervised visionand-language pre-trained models.