TESTA : Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding (2023)

Abstract: Large-scale video-language pre-training has made remarkable strides in advancing video-language understanding tasks. However, the heavy computational burden of video encoding remains a formidable efficiency bottleneck, particularly for long-form videos. These videos contain massive visual tokens due to their inherent 3D properties and spatiotemporal redundancy, making it challenging to capture complex temporal and spatial relationships. To tackle this issue, we propose an efficient method called TEmporal-Spatial Token Aggregation (TESTA). TESTA condenses video semantics by adaptively aggregating similar frames, as well as similar patches within each frame. TESTA can reduce the number of visual tokens by 75% and thus accelerate video encoding. Building upon TESTA, we introduce a pre-trained video-language model equipped with a divided space-time token aggregation module in each video encoder block. We evaluate our model on five datasets for paragraph-to-video retrieval and long-form VideoQA tasks. Experimental results show that TESTA improves computing efficiency by 1.7 times, and achieves significant performance gains from its scalability in processing longer input frames, e.g., +13.7 R@1 on QuerYD and +6.5 R@1 on Condensed Movie.

## introduction video-language modeling aims to learn semantic alignment between video and language in a joint representation space @xcite to facilitate downstream tasks including text-video retrieval, video question answering (videoqa), and video captioning. unlike text, which can be represented concisely as a sequence of words with dense semantics, video input consists of much longer sequences due to its 3d properties and the redundancy in space-time information @xcite . in fact, the number of visual tokens processed by transformer-based models @xcite @xcite @xcite can be over 150× more than text tokens. 2 this poses an efficiency bottleneck for video-language understanding, especially for long-form videos lasting more than 30 seconds @xcite . to encode long videos within limited computing budgets, previous approaches can be broadly categorized into two types: (1) sparse sampling @xcite @xcite . this method reduces the number of visual tokens by sampling very few frames from the raw video. 3 however, sparse sampling sacrifices rich temporal dynamics and storyline information, which limits model performance. (2) offline encoding @xcite . it allows processing more frames within the same computation budgets by constraining the interaction between visual tokens. it first uses an off-the-shelf image encoder @xcite to encode each frame independently, then uses a temporal module to aggregate all the frame features. however, the frame features encoded offline may not be well adapted to downstream tasks in various domains. additionally, the postaggregation mechanism also prohibits the full fusion of frame features @xcite . considering that both sufficient input frames and full temporal-spatial modeling in an end-to-end manner are pivotal for optimal performance, a natural question arises: are there better approaches to achieve efficient video coding without compromising on either of these aspects? in this paper, we propose an efficient method named temporal-spatial token aggregation (testa) inspired by token merging (tome) @xcite . specifically, testa samples input frames densely, but progressively aggregates similar visual tokens during video encoding to reduce the token number and computational overhead. as shown in fig. 1 , our aggregation operates separately in temporal and spatial dimensions, allowing for the merging of similar frames as well as similar patches within each frame. this reduces tome's complexity from o(( building upon testa, we design a pre-trained video-language model with a temporal and spatial token aggregation module in each video encoder block. we evaluate our model on paragraphto-video retrieval and long-form videoqa tasks. when using an equal number of input frames, our model improves computing efficiency by 1.7 times while maintaining comparable performance. when accessing more frames, our model exhibits strong scalability and achieves significant performance gains compared to previous state-of-the-art methods (e.g., +13.7 r@1 on queryd and +6.5 r@1 on condensed movie). ## related work video-language pre-trained models. benefitting from large-scale video-text datasets @xcite and advances in transformer model design @xcite @xcite @xcite , pre-trained video-language models (vidlms) @xcite @xcite have demonstrated impressive performance in video-language understanding tasks. vidlms typically comprise a video encoder and a text encoder, which encode video-text pairs into a shared feature space to learn the semantic alignment between video and language. additionally, a text decoder can be added after the video ## method in this section, we first introduce our videolanguage pre-trained model and its architecture in § 3.1. to improve the efficiency of encoding longform videos, we propose a novel temporal-spatial token aggregation mechanism ( § 3.2). finally, we present the pre-training objectives in § 3.3. ## experiments 4.1 implementation details to pre-train our testa model, we start by initializing it with the blip (12-layer vit-b/16) checkpoint @xcite , with the exception of the temporal attention, which is copied from the spatial attention weights. we use around 5m image-text and video-text pairs from two datasets for pre-training. see appendix a for more details. for downstream fine-tuning, we uniformly sample either 32 or 96 frames, each with a resolution of 224 × 224 pixels (196 patches per frame with a patch size of 16). to achieve approximately a 50% reduction in computation cost, we employ different hyper-parameters for aggregation. specifically, for 96-frame inputs, we set r t to 4 and r s to 8, while for 32-frame inputs, r t is 1 and r s is 12. we use geometry-based aggregation by default since it achieves better performance. please refer to appendix b for more fine-tuning details. ## conclusion in this paper, we present testa, an efficient method for long-form video-language understanding. by aggregating similar frames and patches, testa effectively condenses video semantics and accelerates video encoding. experimental results on paragraph-to-video retrieval and videoqa tasks demonstrate that testa outperforms previous sota methods by a considerable margin.