L ite VL : Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling (2022)

Abstract: Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks. However, the pre-training process is computationally expensive due to the requirement of millions of video-text pairs and the redundant data structure of each video. To mitigate these problems, we propose LiteVL, which adapts a pre-trained image-language model BLIP into a video-text model directly on downstream tasks, without heavy pre-training. To enhance the temporal modeling lacking in the image-language model, we propose to add temporal attention modules in the image encoder of BLIP with dynamic temporal scaling. Besides the model-wise adaptation, we also propose a non-parametric pooling mechanism to adaptively reweight the fine-grained video embedding conditioned on the text. Experimental results on text-video retrieval and video question answering show that the proposed LiteVL even outperforms previous video-language pre-trained models by a clear margin, though without any video-language pre-training.

## introduction the increasing popularity of videos in various social media has aroused the interest in efficient modeling of videos and their connections with other modalities like texts. video-language modeling targets at learning a shared multimodal semantic space for videos and texts to facilitate downstream tasks like text-video retrieval and video question answering (videoqa). previous video-language modeling usually relies on pre-training on a large-scale video-text pair, via video-text contrastive learning @xcite @xcite , video-text matching @xcite @xcite , masked language modeling @xcite and masked frame modeling @xcite ; or extra object detectors to extract fine-grained visual features @xcite . however, both pretraining on video-text pairs and using an off-theshelf detector are computationally expensive and inefficient. inaccurate detection results on limited categories may also lead to inferior performance. in the unimodal video domain, timesformer @xcite fails to pre-train video encoder directly on large-scale video datasets, but obtains good performance by initializing from a pre-trained transformer-based vit @xcite image encoder and training additional temporal attention modules directly on downstream video tasks. similarly, vivit @xcite also takes advantage of the already well-learned spatial visual representation in a pre-trained vit model, and effectively adapts it for video tasks by directly fine-tuning on comparatively small downstream video datasets. inspired by timesformer and vivit, in this paper, we also consider extending an image-language pre-trained model for video-text tasks without pretraining on video-text pairs. this requires us to not only leverage the already-learned alignment between spatial visual information and text in the image-language models, but also capture the additional temporal dependency efficiently. thus we propose a simple yet efficient video-language model litevl, initialized from a recent pre-trained image-language model blip, but with both modelwise and feature-wise enhancement of temporal information. for model-wise enhancement, we propose to explicitly insert temporal attention layers with learnable scalings into the original image backbone, which can be adjusted for each downstream task. for feature-wise enhancement, we design a non-parametric pooling method to learn fine-grained spatial-temporal video features conditioned on the text description. empirical results on various tasks demonstrate that the proposed model, litevl, outperforms previous state-of-the-art methods by a clear margin, even without any video-language pre-training or the usage of object detectors. in particular, litevl achieves 50.8% r1 score on msrvtt-9k dataset in the task of text-video retrieval, and 42.9% accuracy on msrvtt-qa dataset in the task of video question answering. visualizations also demonstrate that our litevl captures important spatialtemporal information with fine-grained video-text alignment. ## method in this section, we propose a method to efficiently extend an image-language pre-trained model to a video-language model, without pre-training on video-text pairs or the use of object detectors. in section 3.1, we first introduce the model architecture. then we propose to enhance the temporal information in both model-wise and featurewise manners. for model-wise enhancement, we propose to insert temporal attention layers with learnable scalings into the original image backbone (section 3.2). for feature-wise enhancement, we design a non-parametric pooling method to learn fine-grained spatial-temporal video features conditioned on the text description (section 3.3). ## experiments in this section, we evaluate the efficacy of the proposed litevl on the text-video retrieval and video question answering (videoqa) tasks. we initialize the weights of litevl from blip @xcite , which uses a vit-b/16 as the image encoder, a bert base as the text encoder with additional crossattention layers for the image-grounded text encoder. we use both blip variants pre-trained on 14m and 129m image-text pairs, respectively, and the corresponding litevl initialized from them are dubbed as litevl s and litevl l , respectively. during fine-tuning, we randomly sample 8 and 16 frames per video for retrieval and videoqa tasks, respectively. while in the inference stage, the frames are uniformly sampled. following previous works @xcite ) and blip's pre-training setting, we resize each of the raw frames to 224×224 before feeding them into the model. for the text-dependent pooling, the temperature τ in eq.( 3 ) is set to 1.0 by default. more detailed training details and hyperparameters are in appendix a. ## conclusion we propose litevl, a video-language model without heavy video-language pre-training or object detectors. litevl inherits the already-learned alignment between the spatial visual information and textual information, from a pre-trained imagelanguage model. then, an extra temporal attention with dynamic temporal scaling is proposed to learn the temporal dynamics in the video. we also introduce a non-parametric pooling method which aggregates video features conditioned on the text description, enabling fine-grained video-language alignment. empirical results show that our litevl outperforms the state-of-the-art methods trained with much more training data.