PEVL : Position-enhanced Pre-training and Prompt Tuning for Vision-language Models (2022)

Abstract: Vision-language pre-training (VLP) has shown impressive performance on a wide range of cross-modal tasks, where VLP models without reliance on object detectors are becoming the mainstream due to their superior computation efficiency and competitive performance. However, the removal of object detectors also deprives the capability of VLP models in explicit object modeling, which is essential to various position-sensitive vision-language (VL) tasks, such as referring expression comprehension and visual commonsense reasoning. To address the challenge, we introduce PEVL that enhances the pre-training and prompt tuning of VLP models with explicit object position modeling. Specifically, PEVL reformulates discretized object positions and language in a unified language modeling framework, which facilitates explicit VL alignment during pre-training, and also enables flexible prompt tuning for various downstream tasks. We show that PEVL enables state-of-the-art performance of detector-free VLP models on position-sensitive tasks such as referring expression comprehension and phrase grounding, and also improves the performance on position-insensitive tasks with grounded inputs. We make the data and code for this paper publicly available at https://github.com/thunlp/PEVL.

## introduction recent progress on self-supervised learning has led to powerful vision-language pre-training (vlp) models that achieve state-of-the-art performance on a wide range of cross-modal tasks @xcite @xcite @xcite . typically, vlp models are first pre-trained on large-scale image-text data to learn universal cross-modal representations, and then fine-tuned to adapt to downstream tasks @xcite . while most traditional vlp models heavily rely on external object detectors to obtain the visual inputs @xcite @xcite , recently there is a growing interest in vlp models that remove the reliance on object detectors due to their superior computation efficiency and competitive performance @xcite @xcite . however, the removal of object detectors also deprives the capability of vlp models in explicit object modeling. the drawback hinders successful handling of vision-language (vl) tasks which are inherently object-centric, where deep understanding of objects and their interactions plays an essential role @xcite @xcite . therefore, it is typically difficult for detector-free foot_0 vlp models to handle various position-sensitive tasks (i.e., tasks that demand explicit object positions as input or output), such as visual commonsense reasoning @xcite , visual relation detection @xcite , referring expression comprehension @xcite and phrase grounding @xcite , which greatly undermines their generality and practicality as foundation models @xcite . for tasks that do not require explicit object modeling, such as visual question answering (antol why is @xmath0 what is the @xmath1. in a preliminary exploration, mdetr @xcite proposes to enhance detector-free vlp models by regressing object positions with transformer decoders, serving position-output tasks such as referring expression comprehension. however, it is still unknown how to deal with various positioninput tasks, such as visual commonsense reasoning and visual relation detection. moreover, during fine-tuning, task-specific classification heads are typically introduced, resulting in a significant gap between pre-training and fine-tuning, which hinders taking full advantage of pre-trained model capabilities in downstream tasks. in this work, we propose pevl that enhances the pre-training and prompt tuning of vlp models with explicit object position modeling. inspired by the recent pix2seq @xcite that casts object detection as a language modeling task, pevl reformulates object positions as discrete tokens, and learns the joint distribution of object positions and language in a unified language modeling framework, as shown in figure 1 . specifically, pevl exploits explicit region-text alignments in existing vl datasets. discretized position tokens are placed after object text tokens to indicate the object locations in both pre-training and prompt tuning: we conduct comprehensive experiments on five vl tasks, including position-output, input and insensitive tasks. experimental results show that through position enhancement, pevl enables stateof-the-art performance of detector-free vlp models on position-sensitive tasks such as referring expression comprehension and phrase grounding, and also improves the performance on positioninsensitive tasks with grounded inputs. our contributions are threefold: (1) we unify the modeling of positions and language in a language modeling framework, which enhances both pre-training and prompt tuning of vlp models. (2) we present a novel ordering-aware objective that retains the ordering of position tokens and avoids the influence of position annotation noise. (3) we conduct comprehensive experiments on five positionsensitive and insensitive vl tasks, which demonstrates the effectiveness of the proposed model. ## preliminary in principle, the pevl framework is orthogonal to vlp architectures and can be built on any vlp models to achieve position enhancement. in this work, without loss of generality, we adopt al-bef @xcite as the model backbone, which is a representative detector-free vlp model that achieves state-of-the-art performance on many vl tasks. we briefly introduce the pre-training and fine-tuning procedure of albef, and refer readers to the original paper for additional details. pre-training. the albef architecture is composed of two unimodal encoders followed by a cross-modal encoder. images and text are first encoded using a vision transformer @xcite and a text transformer @xcite respectively, and then fused with a crossmodal transformer. the model is pre-trained with three tasks, including masked language modeling, image-text contrastive learning and imagetext matching. (1) masked language modeling aims to recover masked text tokens from the crossmodal context. (2) image-text contrastive learning aligns the intermediate unimodal representations of image-text pairs by a contrastive loss. (3) imagetext matching classifies whether an image-text pair is aligned based on the [cls] token of the crossmodal transformer. to alleviate the noise in the pre-training text, a momentum model is maintained based on the moving-average of model parameters to provide pseudo-targets as additional supervision. fine-tuning. during fine-tuning, albef introduces new classification heads or decoders to handle vl tasks, which leads to significant gap from pre-training. the gap hinders taking full advantage of pre-trained capabilities for downstream tasks. moreover, since object positions cannot be explicitly modeled, detector-free vlp models typically struggle on position-sensitive tasks, which greatly undermines their generality and practicality. ## methodology we introduce the pevl framework, including the position reformulation for vl models, and positionenhanced vl pre-training and prompt tuning. ## experiments we evaluate pevl on five popular vl tasks. the models are in base size unless otherwise specified. ## related work position enhancement for vlp. object position modeling underpins a wide range of vl tasks. to deal with position-output tasks, some works @xcite @xcite propose to perform object position prediction using transformer decoders, but are unable to handle various position-input tasks. to explicitly indicate position inputs for vlp models, previous works explored learning region embeddings @xcite , or color-based cross-modal coreferential markers @xcite , but rely on external object detectors. merlot @xcite also proposes to highlight objects in images with colors for position-input tasks. x-vlm @xcite aligns multi-grained concepts in text and image for vlp models. in comparison, pevl supports both position-input and output vl tasks in a unified prompt tuning framework. prompt tuning. prompt tuning for pre-trained language models is in rapid growth in natural language processing @xcite @xcite @xcite @xcite . recently there is also growing interest in prompt tuning vlp models. most existing works prompt tune contrastively pretrained image-text matching models @xcite for recognition tasks @xcite @xcite @xcite . cpt @xcite prompt tunes vlp models with color-based prompts, and achieves promising results for zero-and few-shot tasks. some works perform pre-training and vl tasks using identical transformer decoders in an auto-regressive fashion @xcite @xcite , which avoids the gap between pre-training and tuning, but are typically limited in performance due to the unidirectional architecture. ## conclusion and future work in this work, we present pevl that enhances the pre-training and prompt-tuning of detectorfree vlp models with unified position and language modeling. comprehensive experimental results demonstrate the effectiveness of the proposed model. future works include exploring weakly supervised signals for position and language learning without human annotation.