TRIPS : Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection (2022)

Abstract: Vision Transformers (ViTs) have been widely used in large-scale Vision and Language Pre-training (VLP) models. Though previous VLP works have proved the effectiveness of ViTs, they still suffer from computational efficiency brought by the long visual sequence. To tackle this problem, in this paper, we propose an efficient vision-and-language pre-training model with Text-Relevant Image Patch Selection, namely TRIPS, which reduces the visual sequence progressively with a text-guided patch-selection layer in the visual backbone for efficient training and inference. The patch-selection layer can dynamically compute text-dependent visual attention to identify the attentive image tokens with text guidance and fuse inattentive ones in an end-to-end manner. Meanwhile, TRIPS does not introduce extra parameters to ViTs. Experimental results on a variety of popular benchmark datasets demonstrate that TRIPS gain a speedup of 40% over previous similar VLP models, yet with competitive or better downstream task performance.

## introduction in recent years, vision-language pre-training (vlp) @xcite @xcite @xcite @xcite @xcite @xcite has developed at an astonishing rate and become a prevalent paradigm to tackle vl tasks. traditional vlp models @xcite @xcite utilize pretrained object detectors @xcite @xcite to extract region-based image features but suffer from extensive annotation and expensive computation of object detector training. inspired by the success of the vision transformer (vit) @xcite and its variants @xcite ; wang * corresponding author. @xcite in computer vision field, more recent vlp models @xcite @xcite @xcite have adopted vit as the visual encoder or cross-modal fusion encoder without using region features from the pre-trained object detector. however, these vit-based vlp methods are required to model long visual sequences from highresolution images for good vision understanding, with quadratic computational complexity to the length of the visual sequence. moreover, recent efforts @xcite have also begun to explore vision-language foundation models, which scale up the model and data size. this raises the necessity to decrease the high computational cost of vit-based vlp models. as shown in figure 1 (a), we find that removing the inattentive patch tokens of the image @xcite token will generally not affect the result of visual question answering (vqa) @xcite prediction. based on the observation, we conjecture that not all image tokens in the visual encoder contribute positively to the final prediction result of vlp models, and large numbers of redundant image tokens exist. there have been some recent studies @xcite @xcite focusing on vit model accelerations by reducing unrelated image tokens. however, these methods are specially designed for computer vision tasks (e.g., image recognition) and remove the redundant tokens based on visual semantics, ignoring the aligned knowledge in text modality, and thus are not suitable for vl tasks. as shown in figure 1 (b), without the guidance of text knowledge, directly removing patch tokens based on the image @xcite token will lead to a wrong answer. this observation motivates us to reduce the image tokens by fusing the less informative patch tokens with the guidance of aligned text context. in this work, we propose an efficient vlp model q:what is on the mountain tops? image patches selected by the image [cls] token. ## method in this section, we will first introduce trips with the acceleration module of the text-relevant patchselection layer, and then give the details of the pre-training objectives. ## conclusion we have presented trips, an efficient vlp model with text-relevant image patch selection to progressively reduce the redundant image tokens with text guidance. trips introduces a novel patch selection module to select the text-consistent image tokens in the vision backbone, which preserve the attentive image tokens with text guidance and fuses the inattentive tokens into one token by dynamically computing text-dependent visual attention in an end-to-end way. the experiment shows our method not only decreases the computation cost of vlp but also improves the efficiency of cross-modal fusion due to the reduction of visual sequences, while keeping or even improving the performance of downstream image-text tasks.