DREAM : Improving Video-Text Retrieval Through Relevance-Based Augmentation Using Large Foundation Models (2025)

Abstract: Recent progress in video-text retrieval has been driven largely by advancements in model architectures and training strategies. However, the representation learning capabilities of video-text retrieval models remain constrained by low-quality and limited training data annotations. To address this issue, we present a novel Vi d eo-Text R etrieval Paradigm with R e levance-based A ug m entation, namely dReAm, which enhances video and text data using large foundation models to learn more generalized features. Specifically, we first adopt a simple augmentation method, which generates self-similar data by randomly duplicating or dropping subwords and frames. In addition, inspired by the recent advancement in visual and language generative models, we propose a more robust augmentation method through textual paraphrasing and video stylization using large language models (LLMs) and visual generative models (VGMs). To further enrich video and text information, we propose a relevance-based augmentation method, where LLMs and VGMs generate and integrate new relevant information into the original data. Leveraging this enriched data, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of dReAm over existing methods. Code will be available upon acceptance.

## introduction video-text retrieval (vtr) @xcite @xcite @xcite @xcite @xcite is a fundamental task in visuallanguage understanding @xcite @xcite @xcite @xcite . the recent progress in vtr is mostly driven by powerful pretraining models @xcite @xcite , improved retrieval methods @xcite @xcite , and the newly emerged large-scale video-language benchmark datasets @xcite @xcite . the most widely adopted vtr paradigm @xcite @xcite learns a joint feature space across the visual and textual modalities, where video and text data are directly compared. inspired by the success of clip @xcite , clip4clip @xcite finetunes clip @xcite and investigates three similarity measures for videosentence contrastive learning, with satisfying retrieval performance. subsequently, x-clip @xcite introduces a novel multi-grained contrastive learning framework to further enhance the detailed association between video and text modalities. following these pioneering works, many other methods have also been proposed @xcite @xcite @xcite @xcite @xcite . though different modeling or training techniques have been employed to improve the performance on the modeling side, data issues still exist. for example, most methods are trained using datasets with one-to-one video-text labels, assuming that the video and text data can be wellaligned one-to-one in the same feature space. however, this assumption may not hold tight @xcite for some popular video-text benchmark datasets @xcite @xcite , because in real applications, a single video may correspond to multiple valid sentences, and vice versa. as shown in figure 1 (an example from msr-vtt @xcite ), the basketball video in figure 1 is paired with the text, "various young people play challenging games of basketball", although it can also match with many other semantically similar sentences such as "guy putting the ## method in this section, we present the definition of videotext retrieval and the details of dream, along with three proposed simple but effective augmentation methods as shown in figure 1 . ## experiments benchmarks. to evaluate the proposed dream, we use three representative vtr benchmarks, i.e., msr-vtt @xcite , msvd @xcite , and activitynet @xcite . details are deferred to the appendix due to the limitation of space. evaluation protocols. to evaluate the retrieval performance of our proposed dream, we use recall at rank k (r@k, higher is better), median rank (mdr, lower is better), and mean rank (mnr, lower is better) as retrieval metrics, which are widely used in previous retrieval works @xcite @xcite . implementation details. our baseline (base model) is x-clip @xcite . following @xcite ; @xcite , we use a standard vision transformer @xcite with 12 layers that are initialized with the public clip @xcite checkpoints. ## conclusion in this paper, we proposed a novel video-text learning paradigm, dream, which effectively aligned video and text spaces using generated information conditioned on original data. first, we showed a simple but effective method, self-augmenting, which generated self-similar data without any parameters by randomly duplicating or removing frames and subwords, significantly enhancing representation learning and mitigating overfitting issues commonly observed with current models. second, inspired by the advancement in large language models (llms) and video generative models (vgms), dream employed a novel augmentation method, which augmented data through paraphrasing captions and transferring video styles. last, to enrich video and text data with relevant information, we proposed to augment data with relevance, which encouraged llms and vgms to inject relevant information, and further novel information into the generated data. this method significantly enriched the data pool, contributing to the robustness and depth of the learned representations. finally, comprehensive experiments conducted on several video-text retrieval benchmarks underline the superior performance of dream.