V ideo CLIP - XL : Advancing Long Description Understanding for Video CLIP Models (2024)

Abstract: Contrastive Language-Image Pre-training (CLIP) has been widely studied and applied in numerous applications. However, the emphasis on brief summary texts during pre-training prevents CLIP from understanding long descriptions. This issue is particularly acute regarding videos given that videos often contain abundant detailed contents. In this paper, we propose the VideoCLIP-XL (eXtra Length) model, which aims to unleash the long-description understanding capability of video CLIP models. Firstly, we establish an automatic data collection system and gather a large-scale VILD pre-training dataset with VIdeo and Long-Description pairs. Then, we propose Text-similarity-guided Primary Component Matching (TPCM) to better learn the distribution of feature space while expanding the long description capability. We also introduce two new tasks namely Detail-aware Description Ranking (DDR) and Hallucination-aware Description Ranking (HDR) for further understanding improvement. Finally, we construct a Long Video Description Ranking (LVDR) benchmark for evaluating the long-description capability more comprehensively. Extensive experimental results on widely-used text-video retrieval benchmarks with both short and long descriptions and our LVDR benchmark can fully demonstrate the effectiveness of our method.

## introduction the contrastive language-image pre-training (clip) model @xcite represents a pivotal development in the field of vision-language pre-training. it integrates text and image encoders to align these two modalities through contrastive learning. this methodology has been effectively applied in various applications, such as zero-shot classification @xcite ), text-image retrieval @xcite , and text-to-image generation @xcite . however, one of the notable limitations of clip is its constrained capacity to process extensive textual descriptions, owing to its text encoder's reliance on maximum positional embeddings with length 77. this limitation greatly restricts the length of input text, and existing studies @xcite have also revealed a de facto effective token limit of just around 20. furthermore, the vanilla clip training procedure's emphasis on brief summary texts compels the text/vision encoder to focus predominantly on the main features of the text/visual input, often overlooking smaller, yet potentially critical details. this issue is particularly acute in videos as compared to images, given that videos encapsulate a wealth of details across successive frames, along with additional information such as the sequence and flow of activities, camera movements, etc. in this context, existing video clip models @xcite @xcite ) that employ vanilla clip training methodologies may struggle to accurately capture complex relationships and attributes, due to their reliance on a simplistic "bag of concepts" approach @xcite . to overcome these limitations, enhancing the model's capability to comprehend long descriptions is crucial. longer texts provide a rich tapestry of attributes and interconnections, offering a pathway to significantly improve the model's performance and applicability in more complex scenarios. to this end, we propose videoclip-xl (extra length), to our knowledge, which is the first video clip model with long-description capability. (1) to be specific, recognizing the insufficiency of public datasets containing (video, long description) pairs, we establish an automatic data collection system, designed to aggregate sufficient and high-quality pairs from multiple data sources. we have successfully collected over 2m (video, long description) pairs, denoted as our vild pretraining dataset. (2) we have discovered that existing clip models for long texts @xcite lack the flexibility to dynamically adapt to the distribution changes within high-dimensional feature space. to address this issue, we introduce text-similarity-guided primary component matching (tpcm), a novel approach that enables the model to better learn cross-modal and crosssample relative distances. (3) we claim that there are two attributes that clip models with the longdescription understanding capability should naturally possess: for a given video and its associated descriptions, it should be able to assign a higher score when the description contains i) more rich and precise detailed contexts; or ii) fewer hallucinations with the same level of detail. to this end, we propose two new tasks to model these two attributes namely detail-aware description ranking (ddr) and hallucinationaware description ranking (hdr). they make the video clip model to learn how to correctly rank multiple descriptions with different levels of details and hallucinations. (4) in order to better evaluate video clip models, we further release a long video description ranking (lvdr) benchmark. given each video and the corresponding ground-truth long description (after human correction) sampled from shot2story @xcite , we iteratively modify a certain proportion of correct contents into hallucination in each step. the model is required to correctly rank these descriptions according to their faithfulness. to evaluate the performance of videoclip-xl, we conduct extensive experiments not only on the video & long-description dataset shot2story @xcite , but also on traditional widely-used msr-vtt @xcite , lsmdc @xcite , didemo @xcite , msvd @xcite and @xcite benchmarks, for the text-video retrieval task. moreover, we evaluate videoclip-xl and other representative clip models on our proposed lvdr benchmark. experimental results demonstrate that our method exhibits superior performance compared with state-of-theart competitors. our main contributions are as follows: vision-language datasets. as the capabilities of multimodal models advance, the need transcends traditional fixed-category image datasets such as imagenet @xcite and ci-far10 @xcite . contemporary open-world applications require datasets that encompass both images/videos and their associated text descriptions. common open-world imagelanguage datasets include visual genome @xcite , conceptual-12m @xcite , sbu @xcite , coco @xcite , and laion-5b @xcite . typical video-language datasets comprise msr-vtt @xcite , msvd @xcite , lsmdc @xcite , webvid @xcite , internvid @xcite , and panda-70m @xcite . however, these datasets generally contain only short captions. on the other hand, a few datasets focus on long descriptions. sharegpt4v @xcite is a large-scale dataset with 1.2m long captions for images. shot2story @xcite includes 20k video clips, each with detailed shotlevel captions and comprehensive video summaries. miradata @xcite deals with uncut video segments and features structural long captions. it contains 57.8k video clips across two scenarios: gaming and city/scenic exploration. the average description length in these collections is often orders of magnitude longer than those in previous datasets @xcite . ## methodology in this section, we introduce our automatic data collection system and the resulting video & long-description (vild) pre-training dataset (sect. 3.1), the text-similarity-guided primary component matching (tpcm) technique (sect. 3.2), two new description ranking tasks (sect. 3.3), and the new long video description ranking (lvdr) benchmark dataset (sect. 3.4). ## conclusion in this paper, we propose videoclip-xl, a video clip model with long-description capability. we establish an automatic data collection system to gather our vild dataset, and propose tpcm to better learn the distribution of feature space during pre-training while expanding the long-description capability. we also introduce two new tasks namely ddr and hdr for further understanding improvement. our lvdr benchmark is helpful for evaluating the long-description capability more comprehensively. extensive experimental results have demonstrated the effectiveness of our method. for future research, we plan to refine the pretraining methodology and increase the amount of data and model size for further improvement. we will also attempt to integrate the architecture of cross-encoders and llms into our method.