KD - VLP : Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation (2022)

Abstract: Self-supervised vision-and-language pretraining (VLP) aims to learn transferable multi-modal representations from large-scale image-text data and to achieve strong performances on a broad scope of vision-language tasks after finetuning. Previous mainstream VLP approaches typically adopt a two-step strategy relying on external object detectors to encode images in a multi-modal Transformer framework, which suffer from restrictive object concept space, limited image context and inefficient computation. In this paper, we propose an object-aware end-to-end VLP framework, which directly feeds image grid features from CNNs into the Transformer and learns the multi-modal representations jointly. More importantly, we propose to perform object knowledge distillation to facilitate learning cross-modal alignment at different semantic levels. To achieve that, we design two novel pretext tasks by taking object features and their semantic labels from external detectors as supervision: 1.) Object-guided masked vision modeling task focuses on enforcing object-aware representation learning in the multi-modal Transformer; 2.) Phrase-region alignment task aims to improve cross-modal alignment by utilizing the similarities between noun phrases and object labels in the linguistic space. Extensive experiments on a wide range of vision-language tasks demonstrate the efficacy of our proposed framework, and we achieve competitive or superior performances over the existing pretraining strategies.

## introduction with the success of bert @xcite in language modeling, self-supervised vision-and-language pretraining (vlp) has attracted much interest from ai community, which aims to learn generalizable multi-modal representations from largescale image-text data. combined with a pretrainthen-transfer strategy, it shows great potential in tackling vision and language reasoning tasks, such as image-text retrieval, visual question answering (vqa) and visual entailment @xcite @xcite @xcite . a critical step in such representation learning is to jointly model linguistic entities and visual semantic concepts (e.g., attributes, objects, and relations), as well as their alignment. however, this is particularly challenging due to large discrepancy in visual and language representations (pixels vs words) and lack of entity-level cross-modal correspondence in supervision. to tackle those challenges, most existing approaches @xcite @xcite adopt a two-step pretraining strategy that firstly utilizes off-the-shelf detectors to parse images into a set of object tokens, and then builds a multi-layer transformer to learn visual and language embeddings jointly. in order to facilitate the multi-modal learning, those networks are typically trained via a set of carefully designed bert-like objectives (e.g. image-text matching). despite its promising performance, the two-step strategy suffers from several limitations: 1) limited visual object concepts as the external detectors are trained on a predefined set of object categories; 2) lack of context cues outside of the object regions, which are crucial for complex reasoning tasks; 3) sub-optimal visual representation due to stage-wise training; and 4) computational inefficiency caused by additional detection modules. to overcome those limitations, recent works attempt to learn a joint visual-linguistic representations in an end-to-end manner @xcite @xcite . these methods directly take dense visual features from image grids as inputs to a multi-modal transformer network, and hence do not rely on external object detectors in both pretraining and finetuning stages. such model design significantly simplifies overall network architecture and allows deeper integration between visual and language features. however, using grid-level features makes it difficult to capture object-level visual concepts, which often results in less expressive multi-modal representations and inferior performances in downstream tasks. in this work, we propose a novel object-aware end-to-end (e2e) vlp approach that inherits the strengths of both types of pretraining strategies mentioned above. our core idea, which we name kd-vlp, is to incorporate visual object concepts in the e2e multi-modal learning, which is instantiated by performing knowledge distillation from semantic objects (e.g., from the off-the-shelf detectors) during the pretraining stage. this allows the network to better capture object representations and hence facilitates learning the alignment of linguistic entities and visual concepts. to achieve this, we introduce two novel pretext tasks to perform object knowledge distillation based on a cnn+transformer architecture: an object-based masked vision modeling task for enforcing objectaware feature embeddings, and a phrase-region alignment task for building correspondence between object regions and language entities. specifically, we adopt a typical cnn backbone+multi-modal transformer model for the pretraining. given an image-text pair, the visual backbone firstly computes a set of visual features on the image grid. then a multi-layer transformer takes the visual features and the corresponding text tokens as input to generate their multi-modal embeddings. based on those embeddings, a set of task-specific heads compute the corresponding objectives to train the entire network in an end-to-end fashion. here, in addition to the commonly-used image-text matching and masked language modeling objectives, we develop two object-aware pretext tasks. the first task, object-guided masked vision modeling (omvm), aims to reconstruct the roi features and semantic label of each object (from an external detector) using the surrounding visual context and text description. to facilitate cross-modal alignment, we also develop a knowledge-guided masking strategy, which samples object candidates for reconstruction according to the similarity scores between the noun phrases in the corresponding text and their semantic labels. the second task, phrase-region alignment (pra), aims to further improve cross-modal alignment by matching the above-mentioned phrase-label similarity scores of each phrase with the cross-modal similarity scores between the noun phrase embeddings and object region embeddings. after pretraining, we then transfer the learned multi-modal representations to different downstream vision-language tasks. we perform pretraining on two widely-used indomain datasets: mscoco caption @xcite and visual genome @xcite , and validate the learned multi-modal representations on five well-known visual-language tasks: visual question answering (vqa), imagetext retrieval, nature language visual reasoning (nlvr 2 ), visual entailment (ve) and visual commonsense reasoning (vcr). empirical results show that our method outperforms the state-of-theart end-to-end approaches by a sizeable margin. to better understand our method, we also provide a detailed ablation study and visualization. the contributions of our work are three-fold: ## related work the existing self-supervised vlp approaches can be largely categorized into two groups: the twostep pretraining and the end-to-end pretraining, depending on whether they rely on visual object embeddings as input for the transformer. two-step pretraining firstly employ an off-theshelf object detector to convert an image into a set of object embeddings, and then feed them into a transformer jointly with text embeddings to generate their multi-modal representations. hence their visual feature networks are not optimized during both pretraining & finetuning stage. most of these methods, such as lxmert @xcite ,vilbert @xcite , vl-bert @xcite , unicoder-vl @xcite and uniter @xcite , adopt bert-like objectives to train their networks, which include masked language modeling (mlm), masked vision modeling (mvm) and image-text matching (itm). in addition, villa @xcite develops an advanced adversarial pretraining and finetuning strategy to improve generalization ability. oscar @xcite and vinvl @xcite introduce object labels to bridge different modalities and revisit the importance of visual features. ernie-vil @xcite exploits structured knowledge in the text and constructs scene graph prediction tasks to learn joint representations. unimo @xcite proposes a unified model to leverage large-scale free text corpus, image collections, and image-text pairs simultaneously through a contrastive learning task. despite their strong performances, those methods are limited by the object detector and neglect visual cues outside of object regions, often leading to mistakes in downstream tasks. end-to-end (e2e) pretraining directly feed dense features on image grids from a visual backbone network into a transformer network along with text tokens. as such, both the visual and transformer networks are optimized jointly in an end-to-end manner in the pretraining & finetuning stage. pixel-bert and soho @xcite pioneer the use of the e2e pretraining architecture and propose a novel visual-dictionary masked vision modeling task. e2e-vlp @xcite presents a pretraining framework supervised with additional object detection and image captioning tasks to enhance visual semantics learning. it is worth noting that their object detection pretext task requires millions of bounding boxes annotation, unable to generalize to large-scale image-text corpus. vilt @xcite is the first to unify vision and language with a pure transformer network, which has a simpler structure and enjoys faster inference. however, compared to the twostep methods, they are typically less expressive in terms of object-level concepts and thus suffer from weaker performances on challenging visual reasoning tasks. our method is in line with the e2e pretraining framework. the key difference is that we propose to facilitate learning object-aware multi-modal representations by performing object semantic knowledge distillation. ## conclusion in this paper, we have proposed a novel selfsupervised vlp method that promotes learning object-aware multi-modal representations in an end-to-end framework. our key idea is to perform object knowledge distillation in both semantic and feature space from external detectors in the pretraining stage. in particular, we develop an objectguided masked vision modeling task for distilling external object knowledge, and a phrase-region alignment task for learning better alignment of linguistic entities and visual concepts. compared with prior works, we achieve competitive or superior performance without relying on sophisticated object detectors during model finetuning and test in downstream tasks. where locations within the bounding box fill 1 and others fill 0.