P ro P y: Building Interactive Prompt Pyramids upon CLIP for Partially Relevant Video Retrieval (2025)

Abstract: Partially Relevant Video Retrieval (PRVR) is a practical yet challenging task that involves retrieving videos based on queries relevant to only specific segments. While existing works follow the paradigm of developing models to process unimodal features, powerful pretrained vision-language models like CLIP remain underexplored in this field. To bridge this gap, we propose ProPy, a model with systematic architectural adaption of CLIP specifically designed for PRVR. Drawing insights from the semantic relevance of multi-granularity events, ProPy introduces two key innovations: (1) A Prompt Pyramid, a hierarchical structure that organizes event prompts to capture semantics at multiple granularity levels, and (2) An Ancestor-Descendant Interaction Mechanism built on the pyramid that enables dynamic semantic interaction among events. With these designs, ProPy achieves SOTA performance on three public datasets, outperforming previous models by significant margins. We will release all code and checkpoints to facilitate further research.

## introduction partially relevant video retrieval (prvr) @xcite is a challenging task that retrieves videos based on queries relevant to only specific segments. unlike traditional textto-video retrieval (t2vr) @xcite , which requires the query to match the entire video, prvr better aligns with realworld scenarios -where long videos often consist of multiple events, and users may only be interested in specific segments. this makes prvr more practical and promising for real-world applications. despite significant progress, most prvr methods @xcite @xcite @xcite follow the paradigm of developing models to process extracted unimodal features. while pretrained vision-language models such as clip @xcite have shown remarkable success in t2vr @xcite , their potential remains underexplored for prvr. a recent work qasir @xcite introduces adapters on top of clip to process super-image features. however, this approach does not involve in-depth structural adaptions, leaving clip's capabilities not fully exploited. to bridge this gap, we propose a model with systematic architectural adaption of clip specifically designed for prvr. our approach builds upon recent prompt-based t2vr methods @xcite @xcite , which demonstrate both effectiveness and efficiency by aggregating video semantics through an explicit global token. however, prvr presents unique challenges: videos must be modeled as compositions of multiple events rather than encoded as single vector representations. a naive extension of t2vr approachestreating each segment as an independent sub-video -fails to capture the rich semantic relationships between events. concretely, there are two fundamental types of event relationships that are crucial to model, as illustrated in figure 1 : intra-segment relations representing compositional semantics between events with hierarchical inclusion relationships, and inter-segment relations capturing contextual dependencies between temporally distinct events. the former ones are important in comprehending long events (eat dinner) composed of multiple sub-events (drink), while the latter ones are crucial in scenes where context semantics are required (drink again). modeling these relations and ensuring semantic interactions of relevant events are beneficial for a comprehensive video understanding @xcite . to effectively model the aforementioned event relations and their semantic interactions, we propose propy (interactive prompt pyramid), a novel clip-based architecture for prvr. propy leverages a set of event prompts focusing on segments of varying granularity and organizes them into a prompt pyramid based on the lengths and positions of their segments. this hierarchically structures a video into multi-granularity events. to account for the distinction between intra-segment and inter-segment relations, we design an ancestor-descendant interaction mechanism, which facilitates direct interactions for intra-segment relations and indirect interactions for inter-segment relations. specifically, an ancestor-descendant relationship is established between two event prompts when their governed segments exhibit an inclusion relationship. direct interactions are permitted only for intra-segment events, while inter-segment interactions are conducted indirectly through upper-level event prompts. with these carefully designed architectures and mechanisms, propy achieves sota performance on three challenging datasets, demonstrating the superiority of our method. overall, our contributions can be summarized as follows: partially relevant video retrieval addresses the task of retrieving videos based on queries relevant to partial segments @xcite . current prvr approaches @xcite @xcite @xcite @xcite predominantly adopt the multiple instance learning (mil) paradigm @xcite , employing coarse-fine two-branch architectures to model multiple events during both training and inference. while some recent works @xcite have incorporated pretrained vision-language models, they primarily utilize them for basic feature extraction without architectural innovations, or source for feature distillation @xcite . qasir @xcite , the most similar work to ours, merely introduces adapters on top of clip to process super-image features while leaving the core clip layers unchanged. we argue that such surface-level modifications are insufficient to fully leverage clip's capabilities for the prvr task. ## methodology we formally define the prvr task: given a set of videos the prvr task aims to retrieve videos with queries t i relevant only to certain segment m i j : ## conclusion we propose propy, the first in-depth clip-based model for prvr. by considering both intra-event and inter-event relations of video events, we design an interactive prompt pyramid architecture to extract multi-granularity event features and an ancestor-descendant interaction mechanism to ensure sufficient semantic interactions. extensive experiments demonstrate the superiority and generalizability of propy.