MS - DETR : Natural Language Video Localization with Sampling Moment-Moment Interaction (2023)

Abstract: Given a text query, the task of Natural Language Video Localization (NLVL) is to localize a temporal moment in an untrimmed video that semantically matches the query. In this paper, we adopt a proposal-based solution that generates proposals (i.e. candidate moments) and then select the best matching proposal. On top of modeling the cross-modal interaction between candidate moments and the query, our proposed Moment Sampling DETR (MS-DETR) enables efficient moment-moment relation modeling. The core idea is to sample a subset of moments guided by the learnable templates with an adopted DETR framework. To achieve this, we design a multi-scale visual-linguistic encoder, and an anchor-guided moment decoder paired with a set of learnable templates. Experimental results on three public datasets demonstrate the superior performance of MS-DETR.

## introduction natural language video localization (nlvl) aims to retrieve a temporal moment from an untrimmed video that semantically corresponds to a given language query, see fig. 1 for an example. this task is also known as temporal sentence grounding in video, and video moment retrieval. as a fundamental video-language task, it has a wide range of applications, such as video question answering @xcite @xcite , video retrieval @xcite @xcite , and video grounded dialogue @xcite . generally speaking, in nlvl models, a video is first split to a sequence of many small fixed-length segments. video features are then extracted from these segments to interact with the text query. conceptually, each video segment can be viewed as a ground-truth query he stops playing and continues talking. moment candidate #1 moment candidate #2 figure 1: an nlvl example with query and ground truth video moment. two moment candidates with similar video features are also highlighted in light and dark green colors. to generate proposals, some methods enumerate all possible moment candidates via pre-defined anchors. anchors are reference start/end positions along the video. fig. 2 shows three 2d-map examples. each cell in a 2d-map corresponds to a candidate moment defined by its start/end time along the two axes. some other methods produce moment candidates with a proposal generator guided by text query and then refine them independently. the interaction between text and video is mainly modeled between text and video moments; each moment is characterized by the video segments that compose it. very few studies have considered moment-moment interaction. consequently, it is challenging to discriminate among moments if there are multiple moments that all demonstrate high level of semantic matching with the text query. for instance, the two candidate moments in fig. 1 have very similar video content and share similar semantic correspondence with the query. start time end time end time end time start time start time moment sampling interaction full global interaction local interaction reference moment interacted moment learnable template proach for its capability of cross-modal interaction at both segment level and moment level. we propose ms-detr to facilitate effective text-moment alignment and efficient moment-moment interaction. for text-moment alignment, we devise a multi-scale vision-language transformer backbone to conduct segment-word and segment-segment interactions at different segment scales. for momentmoment interaction, our main focus is on which moments should be sampled for interaction, due to the large number of possible pairs. recall that a moment is a span of segments. let o(n ) be the magnitude of segment space; the magnitude of moments is o(n 2 ). then moment-moment interaction has a space of o(n 4 ). in practice, not every pair of moments are relevant to each other, and are needed to be discriminated for a given query. existing methods @xcite @xcite ) mainly rely on a strong assumption that only the overlapping or adjacent moments are more likely to be relevant, i.e., moment locality. an example of moment locality is shown in fig. 1 , where two adjacent candidate moments share high level of visual similarity. the local interaction strategy is illustrated in fig. 2 , where the reference moment only interacts with the surrounding moments in the 2d-map. however, not all relevant moments are overlapping or located close to each other. following the example in fig. 1 , if the person plays saxophone again in the later part of the video (not showing for the sake of space), and the query becomes "he plays saxophone again", then there will be at least two highly relevant moments for playing saxophone, separated by his action of talking in between. to correctly locate the answer, the model needs to understand that "again" refers to the second moment of playing saxophone. this calls for a better way of sampling moments for efficient moment-moment interaction, to avoid the full global interaction as shown in fig. 2 . the proposed ms-detr samples moments for interaction using learnable templates and anchors, illustrated in the third 2d-map in fig. 2 . we design an anchor-guided moment decoder to interact and aggregate moment features from the encoder in an adaptive and progressive manner. a fixed number of learnable templates paired with dynamic anchors are used to match the moment content and its location. here, the templates are used to match video content in a moment, and anchors specify the reference start/end positions of the moment because multiple moments may share similar visual features. we then revise the anchors based on the predictions from the last decoder block in an iterative manner. we remark that our method has no assumption on moment locality: the moments can be scattered in diverse locations of the video. our key contributions are threefold. first, we propose a novel multi-scale visual-linguistic encoder (section 4.1) to align textual and video features as well as to aggregate language-enhanced semantics of video frames, in a hierarchical manner. second, we introduce a new anchor-guided moment decoder (section 4.2) to decode learnable templates into moment candidates, in which we propose an anchor highlight mechanism to guide the decoding. third, we conduct extensive experiments (section 5) on three benchmark datasets: activitynet captions, tacos, and charades-sta. our results demonstrate the effectiveness of the proposed ms-detr. ## related work we first briefly review existing nlvl approaches and highlight the differences between our work and other proposal-based solutions. next, we briefly introduce object detection to provide background for the concept of learnable templates. natural language video localization. nlvl was first introduced in @xcite , and since then a good number of solutions have been proposed @xcite . as aforementioned, existing methods can be largely grouped into proposal-based and proposal-free methods. proposals, or candidate moments, can be either predefined @xcite or computed by proposal generator @xcite . proposal-free methods output time span @xcite @xcite or timestamps @xcite @xcite directly on top of video tokens, without considering the notion of candidate moments. most proposal-based methods conduct multimodal interaction between video segments and text, then encode moments from the segment features. typically there is no further interactions among moments. 2d-tan @xcite is the first to demonstrate the effectiveness of moment-level interaction. however, 2d-tan assumes moment locality and only enables local interactions among moments as shown in fig. 2 . however, similar moments requiring careful discrimination may be scattered all over the video. this motivates us to go beyond the moment locality assumption and propose moment sampling for interaction, which is a key difference and also a contribution of our work. in this paper, we adapt the concept of learnable templates from detr framework to achieve dynamic moment sampling. detr was originally introduced for object detection in computer vision (cv), to be briefed shortly. most similar to our work is @xcite , which also uses learnable templates. however, their work directly adopts learnable templates without any adaption to the specific requirements of nlvl. for instance, the answer moment in nlvl needs to match the given text query, whereas in object detection, there is no such requirement. we bridge the gap between nlvl and object detection by introducing a hierarchical encoder and a decoder with an anchor highlight mechanism. these designs greatly improve performance and unveil the potential of detr for nlvl. at the same time, these designs also make our model much different from the original detr. ## problem formulation we first present how to map video and text into features, and then define nlvl in feature space. the m words are encoded with pre-trained word embeddings as @xmath0. given the video and text query in their encoded features (v, l), the task of nlvl is to localize the timestamp pair (t s , t e ), the start and end timestamp, of the video moment that matches the query. note that, due to the uniform split to segments, there is a correspondence between t s and t e of the original video and the segment ids in the segment sequence. ## method the main architecture of the proposed ms-detr is depicted in fig. 3 . illustrated in the feature extraction part, given visual features @xmath1, we first project them into a unified dimension d using single layer ffn and decorate them by adding positional encoding, respectively. the linearly projected visual features ## experiments we evaluate ms-detr against baselines on three public benchmarks: activitynet captions @xcite , tacos @xcite , and charades-sta @xcite . the three datasets cover videos from different domains and lengths (see appendix a.1 for video distributions and train/dev/test splits). following prior work @xcite , we adopt "r@n, @xmath2. r@n, @xmath3, where "hitting" means an overlapping with @xmath4. miou denotes the average iou over all test samples. we set @xmath5.3, 0.5, 0.7}. in our comparison and discussion, we mainly focus on µ = 0.7 as large iou means high-quality matching. ## conclusion in this paper, we adapt detr framework from object detection to nlvl. with the proposed ms-detr, we are able to model moment-moment interaction in a dynamic manner. specifically, we design a multi-scale visual-linguistic encoder to learn hierarchical text-enhanced video features, and an anchor-guided moment decoder to guide the attention with dynamic anchors for iterative anchor refinement. the promising results on three benchmarks suggest that moment-moment interaction for nlvl can be achieved in an efficient and effective manner.