CONE : An Efficient CO arse-to-fi NE Alignment Framework for Long Video Temporal Grounding (2023)

Abstract: This paper tackles an emerging and challenging problem of long video temporal grounding (VTG) that localizes video moments related to a natural language (NL) query. Compared with short videos, long videos are also highly demanded but less explored, which brings new challenges in higher inference computation cost and weaker multi-modal alignment. To address these challenges, we propose CONE, an efficient COarse-to-fiNE alignment framework. CONE is a plug-and-play framework on top of existing VTG models to handle long videos through a sliding window mechanism. Specifically, CONE (1) introduces a query-guided window selection strategy to speed up inference, and (2) proposes a coarse-to-fine mechanism via a novel incorporation of contrastive learning to enhance multi-modal alignment for long videos. Extensive experiments on two large-scale long VTG benchmarks consistently show both substantial performance gains (e.g., from 3.13 to 6.87% on MAD) and state-of-the-art results. Analyses also reveal higher efficiency as the query-guided window selection mechanism accelerates inference time by 2x on Ego4D-NLQ and 15x on MAD while keeping SOTA results. Codes have been released at https://github.com/houzhijian/CONE .

## introduction video temporal grounding @xcite aims to locate specific moments in an untrimmed video relevant to a textual user query. this is a crucial task in multi-modal video understanding and can be applied to many practical applications such as video retrieval @xcite , video editing, and video question answering @xcite . temporal grounding for long videos is especially highly-demanded and emerging due to the 0 s fine level outdoor 21 s 26 s 21 s 26 s : what object was the woman in the room holding? answer figure 1: an example of long video temporal grounding, which requires coarse-to-fine multi-modal alignment. to address these challenges, we propose cone, a coarse-to-fine alignment framework for long video temporal grounding. first, we slice the arbitrary long video into candidate windows. then, we employ a query-guided window selection strategy for efficiency and further introduce a coarse-tofine mechanism for effectiveness. specifically, the query-guided window selection strategy efficiently reduces the sizeable window number of the long video to a modest amount via efficient alignment score computation. moreover, the coarse-to-fine alignment mechanism consists of three modules to gradually align the multi-modal inputs via multiscale granularity: (1) window (coarse granularity) selection to reliably select semantically relevant candidate windows; (2) window (coarse)-proposal (fine) joint contrastive learning to generate candidate proposals considering both inter-window and intra-window relations; (3) proposal (fine granularity) ranking to further accurately sort out the perfect proposal. notably, we incorporate contrastive learning into this mechanism. on the one hand, we utilize the powerful multi-modal alignment ability of contrastive vision-text pre-trained models (e.g., clip), which computes matching scores between video frames and textual query for both window selection and proposal ranking stages. on the other hand, we select a contrasting negative window and design an inter-window contrastive loss during training for the joint learning stage. with this coarse-to-fine design, cone has the following advantages: (1) higher efficiency in handling long video inputs with less temporal information loss; (2) more accurate multi-modal alignment with less contextual information loss. we evaluate cone on two large-scale long video grounding benchmarks and achieve both the state-ofthe-art results and significant performance boosts (3.13%→6.87% on mad, and 10.46%→13.46% on ego4d-nlq in terms of r1@@xmath0.3). further analysis shows that the window selection strategy shortens the inference speed by 2x for ego4d-nlq and 15x for mad compared to inference on every window, without sacrificing its performance. contributions our work presents two major contributions to the long video temporal grounding field. (1) we propose a novel coarse-to-fine alignment framework that utilizes a pipeline of {window slicing and selection, proposal generation and ranking}, resulting in state-of-the-art performance and high efficiency on two representative benchmarks. (2) we introduce a novel approach for incorporating contrastive learning into multi-modal alignment. furthermore, other studies have focused on different issues. for example, naq @xcite addresses data scarcity with an effective data augmentation strategy, while denet @xcite tackles the ground-truth bias problem with a de-coupling and de-bias strategy. there are also other related tasks involving language grounding with video corpus input @xcite or spatial-temporal output @xcite , but they fall outside the scope of this paper. long-form video modeling. long-form video modeling is an emerging topic recently investigated in (multi-modal) video understanding, including classification, segmentation, localization, and retrieval. the common challenges of long videos are modeling long-range temporal dependency, efficiency issues, and accurate multi-modal alignment (if language is involved). to ease long-range dependency issues, some works explore either feature memory banks @xcite , tracked object-level representations @xcite , bayesian non-parametric model @xcite , or structured state-space sequence layers (islam and bertasius, 2022). to alleviate efficiency issues, other works @xcite exploit dense audio sampling as additional information to enable sparse video frame sampling. in contrast, cone mitigates both efficiency issues via query-guided window selection strategy and multi-modal alignment issues via novel contrastive learning incorporation. contrastive learning. contrastive learning has been widely studied in vision @xcite , language @xcite , and multi-modal fields @xcite @xcite . some recent vtg works also attempt to adopt contrastive learning. they mainly focus on frame-level contrastive loss via hard negative mining @xcite or contrasting ground-truth frame with non-ground-truth frame @xcite . in contrast, we further propose window-level contrastive learning to repel the negative windows. moreover, cone also jointly incorporates contrastive vision-text pre-trained models for accurate alignment, because those contrastive models show strong multi-modal alignment ability @xcite derived from pre-training large-scale vision-text pairs. ## task definition since never-ending video streams in real applications have a higher demand for long videos, in this paper, we study the task of video temporal grounding (vtg) in a more challenging setting with long video inputs. taking a natural language (nl) query q and a long video v as the inputs, the task of vtg requires the system to locate the matched moment m from the video v relevant to the query q. formally, video @xmath1, v 2 , . . . , v lv ] is a sequence of uniformly sampled video frames, and l v denotes the length of the sampled frames. the nl query @xmath2, q 2 , . . . , q lq ] is a sequence of tokens with sentence length l q . the moment m is a subsequence of v that is relevant to q. the long-form video setting is more challenging because of the natural demands for more computation and time to process the entire video. moreover, the accurate multi-modal alignment between each v i and q is also harder when l v increases. ## approach we present the proposed cone for long video temporal grounding. as shown in fig. 2 , we first slice the long video into several fixed-length video windows via a sliding window mechanism ( § 4.1). then, we propose a coarse-to-fine mechanism for efficient and effective multi-modal alignment. at the coarse-grained window level, we introduce a query-guided window selection strategy ( § 4.2) to accelerate inference and select semantic relevant windows. at the window-proposal joint learning level, we rely on the existing vtg work for short videos to generate reliable candidate proposals and conduct both intra-window and proposed interwindow contrastive learning ( § 4.3) to assign each proposal score. at the fine-grained proposal level, we further accurately rank candidate proposals with a fine-grained ranking ( § 4.4) mechanism. ## experiments we conduct experiments to explore the effectiveness of cone from the following aspects: (1) model comparison with sota methods ( § 5.3); ## conclusions we present cone , a coarse-to-fine alignment framework for long video temporal grounding. cone jointly achieves state-of-the-art performances on two benchmarks and high efficiency while keeping high frame sample rate. through our experiments, we show the proposed coarse-tofine mechanism via contrastive learning plays an important role in boosting performance. regarding efficiency, the introduced query-guided window selection strategy largely accelerates inference by 2x and 15x on ego4d-nlq and mad benchmarks, respectively, with even slight performance gains. since cone can be generalized with different existing proposal-based models, we hope it can be used to improve the model efficiency and performance for temporal grounding on long videos.