ELIOT : Zero-Shot Video-Text Retrieval through Relevance-Boosted Captioning and Structural Information Extraction (2025)

Abstract: Recent advances in video-text retrieval (VTR) have largely relied on supervised learning and fine-tuning. In this paper, we introduce , a novel zero-shot VTR framework that leverages off-the-shelf video captioners, large language models (LLMs), and text retrieval methods—entirely without additional training or annotated data. Due to the limited power of captioning methods, the captions often miss important content in the video, resulting in unsatisfactory retrieval performance. To translate more information into video captions, we first generates initial captions for videos, then enhances them using a relevance-boosted captioning strategy powered by LLMs, enriching video descriptions with salient details. To further emphasize key content, we propose structural information extraction, organizing visual elements such as objects, events, and attributes into structured templates, further boosting the retrieval performance. Benefiting from the enriched captions and structuralized information, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of over existing fine-tuned and pretraining methods without any data. They also show that the enriched captions capture key details from the video with minimal noise. Code and data will be released to facilitate future research.

## introduction video-text retrieval (vtr) @xcite @xcite @xcite @xcite aims to retrieve the corresponding video or text given the query in another modality. recent years have witnessed the rapid development of vtr with the support from powerful pretraining models @xcite @xcite , improved retrieval methods @xcite @xcite , and video-language datasets construction @xcite . however, it remains challenging to precisely match video and language due to the raw data being in heterogeneous spaces and the use of modality-specific encoders. the most popular paradigm in vtr @xcite @xcite firstly learns a joint feature space across modalities and then compares representations in this space. however, with the discrepancy between different modalities and the design of modality-independent encoders, it is challenging to directly match representations of different modalities generated from different encoders @xcite . on the other side, pioneering works @xcite convert images into captions for better presentation learning on image-language tasks, demonstrating that captioners can mitigate modality discrepancy. in this work, we propose eliot, a zeroshot generative video-to-text retrieval framework. eliot transforms raw videos into enriched generative identifiers by employing a distillationenhanced generative approach. drawing from recent advancements in identifier generation (e.g., titles, substrings, multiview representations) and inspired by distillation-enhanced generative retrieval (dgr), our method incorporates the structural benefits of multiview generative identifiers while addressing the challenges of modality alignment. key to our approach is a novel relevance-boosted captioning mechanism that generates comprehensive textual descriptions for videos. this process ensures that important details such as objects, events, and attributes are captured. to refine these captions, we employ a distilled generative identifier extraction method, replacing traditional structural extraction with a generative paradigm that encodes semantic and contextual cues from videos into identifier representations. by distilling fine-grained ranking knowledge from a teacher model into the generative process, eliot enhances the quality of identifiers without additional training. finally, to evaluate the effectiveness of our proposed zero-shot eliot, we conducted experiments on three representative video-text benchmarks @xcite @xcite . results show that eliot outperforms previous methods, including fine-tuning methods and few-shot methods benefiting from relevance-boosted captioning and structural information extraction. in summary, our contributions are as follows: video-text retrieval, which involves cross-modal alignment and abstract understanding of temporal images (videos), has been a popular and fundamental task of language-grounding problems @xcite @xcite . most of the existing video-text retrieval frameworks @xcite @xcite @xcite @xcite focus on learning powerful representations for video and text and extracting separated representations. for example, in @xcite , videos and texts are encoded using convolutional neural networks and a bi-gru @xcite ) while mean pooling is employed to obtain multi-level representations. mmt @xcite uses a cross-modal encoder to aggregate features extracted by temporal images, audio, and speech for encoding videos. following that, mdmmt @xcite further utilizes knowledge learned from multi-domain datasets to improve performance empirically. further, mil-nce @xcite adopts multiple instance learning and noise contrastive estimation, addressing the problem of visually misaligned narrations from uncurated videos. recently, with the success of self-supervised pretraining methods @xcite @xcite , vision-language pretraining @xcite @xcite on large-scale unlabeled crossmodal data has shown promising performance in various tasks, e.g., image retrieval @xcite , image captioning @xcite , and video retrieval @xcite . recent works @xcite @xcite @xcite @xcite have attempted to pretrain or fine-tune video-text retrieval models in an end-to-end manner. clipbert @xcite , as a pioneer, proposes to sparsely sample video clips for end-to-end training to obtain clip-level predictions and then summarize them. frozen in time @xcite uses end-to-end training on both image-text and video-text pairs data by uniformly sampling video frames. clip4clip @xcite finetunes models and investigates three similarity calculation approaches for video-sentence contrastive learning on clip @xcite . further, ts2-net @xcite proposes a novel token shift and selection transformer architecture that adjusts the token sequence and selects informative tokens in both temporal and spatial dimensions from input video samples. while the mainstream of vtr models @xcite focuses on fine-tuning powerful image-text pre-trained models, on the other side, as a pioneer, @xcite propose to use large language models (llms) for zero-shot video question answering. zero-shot cross-modal retrieval. with the huge success of pretrained visual-language model @xcite , zero-shot crossmodal retrieval has attracted more and more research interest recently. due to the powerful representation learning ability in image and text domains, clip @xcite achieves satisfying zero-shot retrieval performance on several representative image-text retrieval benchmarks @xcite ods. on the other side, benefiting from large-scale video-text benchmarks @xcite @xcite , video-language pre-trained models @xcite @xcite @xcite @xcite also achieve satisfying zero-shot video-text retrieval results. in this paper, inspired by these pioneering works, to explore zero-shot video-text retrieval, we step forward and propose a simple but effective zeroshot video-text retrieval method, eliot, by utilizing off-the-shelf captioning, large language models, and text retrieval methods. ## eliot -zero-shot video text retrieval in this section, we present the details of our proposed method, eliot. specifically, we first generate captions for videos using video caption generation methods. then, to cover most of the details in videos, with our proposed relevance-boosted caption generation, we obtain a detailed caption containing almost all the details. finally, we propose the structural information extraction to emphasize important information in the captions for better video-text retrieval performance. the whole procedure and figure are summarized in figure 1. ## conclusion in this paper, we present an innovative zero-shot framework, eliot, which revolutionizes videotext retrieval by capitalizing on existing captioning methods, large language models (llms), and text retrieval techniques. by sidestepping the need for model training or fine-tuning, our framework offers a streamlined approach to retrieval. to overcome the shortcomings of traditional captioning methods, we propose a groundbreaking relevanceboosted caption generation technique that incorporates llms' generated information into video captions. moreover, our introduction of structural information extraction further enhances retrieval performance by highlighting key visual tokens. through extensive experimentation across diverse benchmarks, we demonstrate the superior efficacy of eliot compared to conventional fine-tuned and pretraining methods, even in the absence of training data. limitations in the future, it would be interesting to explore more detailed methods for zero-shot video-text retrieval, such as incorporating the audio modality and corresponding off-the-shelf foundation models. moreover, as a pioneering work, our work mainly focuses on establishing the paradigm. it would be great if we could explore more text retrieval methods, video captioning methods, and llms for relevance-boosted caption generation. language-based semantic alignment. in the twelfth international conference on learning representations. linchao zhu and yi yang. 2020. actbert: learning global-local video-text representations. in 2020 ieee/cvf conference on computer vision and pattern recognition, cvpr 2020, seattle, wa, usa, june 13-19, 2020, pages 8743-8752. computer vision foundation / ieee.