RAP : Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter (2024)

Abstract: Text-Video Retrieval (TVR) aims to align relevant video content with natural language queries. To date, most of the state-of-the-art TVR methods learn image-to-video transfer learning based on the large-scale pre-trained vision-language models (e.g., CLIP). However, fully fine-tuning these pre-trained models for TVR incurs prohibitively expensive computation cost. To this end, we propose to conduct efficient text-video Retrieval with a salient-and-correlated AdaPter (RAP), i.e., fine-tuning the pre-trained model with a few parameterized layers. To accommodate the text-video scenario, we equip our RAP with two indispensable characteristics including temporal sparsity and correlation. Specifically, we propose a low-rank modulation module to refine the per-image features from frozen CLIP backbone, which accentuates silent frames within the video features while alleviating temporal redundancy. Besides, we introduce an asynchronous self-attention mechanism which firstly selects top responsive visual patch and augments the correlation modeling between them with learnable temporal and patch offsets. Extensive experiments on four TVR datasets demonstrate that our RAP achieves superior or comparable performance compared to the fully fine-tuned counterpart and other parameter-efficient finetuning methods.

## introduction text-video retrieval (tvr) @xcite @xcite @xcite @xcite is a pivotal task in the realm of multimodal research, which aims to find the most relevant video content within a repository in response to the text query, and vice versa. with the rapid progress in large-scale image-text pre-training @xcite @xcite ., * equal contributions. † corresponding author. 2021), current research focuses on how to transfer pre-trained image-text models (e.g., clip @xcite ) to the video-text domain. however, fully fine-tuning the video model is computationally expensive and may have the risk of overfitting. to alleviate this dilemma, parameter-efficient fine-tuning (peft) stemmed from natural language processing @xcite @xcite has also aroused extensive research interest in the field of computer vision @xcite and cross-modal learning @xcite . recently, some exploratory work @xcite @xcite has also attempted to introduce peft into tvr. these methods, however, simply introduce existing peft algorithms tunable parameters (mb in log scale) r@1 of text-to-video retrieval figure 2 : text-to-video retrieval performance on msr-vtt dataset. marker sizes are proportional to the number of tunable parameters. @xcite @xcite without considering the inherent characteristics of video data. to this end, we argue that an ideal peft method for vtr should be equipped with two characteristics: 1) temporal sparsity: as shown in figure 1 , the video data inherently contains lots of redundancies or repetitions in the temporal perspective. the visualized frame-by-frame embedded clip features are over-smooth, resulting in the loss of important details or nuances within the video data. in contrast, the video feature adapted from pre-trained clip should capture the most informative frames, allowing for a more sparse representation. 2) temporal correlation: the desired video adapter is supposed to incorporate the dependencies and relationships between consecutive frames, especially when dealing with actions or events that happen in several frames, as the features can encapsulate the evolving context over time. for example in figure 1 , the query sentence contains two entities including dog and bird. given the query patch ( in frame #3), we visualize the similarity distribution within the other patches. as in this example, vanilla self-attention can only attend to the dog instance while the other bird instance is overlooked. in the realm of video processing and analysis, the temporal dimension often contains redundancies due to the inherent correlation between adjacent frames. this redundancy can lead to inefficiencies in computational resources and storage when dealing with large-scale video data. therefore, there is a need to extract meaningful and informative features while reducing temporal redundancy. to alleviate these aforementioned issues, we propose an efficient text-video retrieval framework with sparse-and-correlated adapter (dubbed as rap). our proposed rap not only streamlines the trainable parameters, enhancing efficiency in computational resources, but also tailors the architecture to adeptly capture and model the nuanced temporal characteristics of video data. to achieve temporal sparsity, we propose a low-rank modulation (lorm) module to refine the pre-trained clip feature @xcite on the principle of redundancy reduction and essential information extraction. this design stems from a simple hypothesis that the change in temporal weights resides on a low intrinsic rank @xcite . therefore, we introduce layer-wise low-rank scale parameters and shift parameters, which could be considered as variance and mean to modulate the clip feature. specifically, both scale and shift parameters are instantiated by the multiplication of two low-rank trainable matrices. these parameters are input-independent and therefore more flexible. lorm allows us to calibrate the video features to highlight salient frames and mitigate temporal redundancy. for temporal correlation modeling, we replace vanilla self-attention with the proposed asynchronous self-attention (asa), which introduces temporal dynamics among video frames to capture temporal relationships. since the attention computing in pre-trained clip is constrained within each frame feature, it is challenging to apply to the video domain due to the temporally dynamic nature of video frames. previous methods employ either temporal transformer @xcite @xcite or 3d convolution networks @xcite to encode temporal dependencies. instead of introducing additional modules, we propose an asynchronous selfattention that only warps partial patch tokens in a parameterized way. firstly, for each frame, we filter semantically significant patches via a parameterfree text-conditioned selection mechanism. specifically, we compute the similarities between patch features and the corresponding sentence and select the patches with the highest responses. secondly, each selected patch within the current frame is dynamically warped to attend to the temporally related patches in other frames. the proposed asynchronous self-attention empowers the flexibility in capturing correlations between video frames at the fine-grained patch level. overall, the main contributions of this work are: text-video retrieval. tvr @xcite @xcite @xcite @xcite @xcite is a fundamental research topic in the video-language domain which aims to retrieval the relevant video/text based on the given text/video query. the pioneer works @xcite rely on pre-extracted features from frozen video and text encoders. to facilitate the end-to-end training, clip-bert @xcite proposes a sparse sampling strategy for efficient text-video training. with the great success of large-scale image-text pretraining model clip @xcite , the majority of the state-of-the-art tvr methods @xcite @xcite @xcite focus on transferring the powerful clip encoder to the video-text domain by designing various cross-modal alignment strategies. as the first attempt, clip4clip @xcite employs mean-pooling or transformer to aggregate video features and conduct coarse-grained (video-sentence level) contrastive alignment. instead of using the text-agnostic aggregation manner, x-clip @xcite proposes to aggregate video representations conditioned on the text's attention weight and conduct the multi-grained contrastive learning at the frame-word, video-sentence, video-word and sentence-frame levels. for more comprehensive alignment, ucofia @xcite unifies the coarse-grained and fine-grained alignment to capture both the high-level and lowlevel correspondence between text and video. most of the current tvr methods follow the fully fine-tuning paradigm. this scheme, however, is computation-intensive and may have the risk of overfitting. besides, additional temporal modeling models are required to bridge the image and video gap. in this paper, we propose rap which conducts parameter-efficient fine-tuning for tvr which provides a more computationally efficient and potentially more robust approach. besides, the tunable parameters in our rap also bear the responsibility for temporal modeling, thus eliminating the need for external temporal modules. parameter-efficient transfer learning. peft @xcite @xcite @xcite is firstly introduced in the nlp domain to reduce the number of trainable parameters while maintaining the comparable performance with the fully fine-tuning setting. inheriting the merit from nlp, peft in computer vision @xcite @xcite also gained extensive research attention. vpt @xcite follows the prompt tuning strategy by introducing the task-specific learnable prompts on the vision transformer. to be more compatible with vision tasks, convpass @xcite introduces the inductive bias of convolutional layers by reconstructing the spatial structure of the token sequence via convolution operations. vl-adapter @xcite pioneeringly benchmarks different types of peft techniques including adapter @xcite , hyperformer @xcite , and compacter @xcite in the multitask setting. there also exist several works @xcite @xcite @xcite @xcite focusing on the imageto-video transfer learning. based on the pre-trained clip model, these methods either introduce temporal convolution @xcite or transformer @xcite in sequential @xcite or parallel @xcite ways. however, they overlook the inherent temporal structure of video data while our rap pinpoints two key issues in video feature modeling and generate more representative video features. the schematic illustration of our rap is illustrated in figure 3 . in sec. 3.1, we first present the preliminaries of rap including the video and text feature embedding. then we describe the proposed low-rank modulation and the asynchronous self-attention in sec. 3.2 and sec. 3.3, respectively. ## conclusions in this work, we present rap to efficiently transfer the pre-trained clip model to tvr. to accommodate the inherent video structure and the crossmodality setting, we introduce a low-rank modulation module to achieve the frame-wise sparse representation and an asynchronous self-attention module to enhance the cross-frame correlations. extensive experiments illustrate that rap achieves comparable or even better performance than previous arts and the fully fine-tuned counterpart.