Natural Language Video Localization with Learnable Moment Proposals (2021)

Abstract: Given an untrimmed video and a natural language query, Natural Language Video Localization (NLVL) aims to identify the video moment described by query. To address this task, existing methods can be roughly grouped into two groups: 1) propose-and-rank models first define a set of hand-designed moment candidates and then find out the best-matching one. 2) proposal-free models directly predict two temporal boundaries of the referential moment from frames. Currently, almost all the propose-and-rank methods have inferior performance than proposal-free counterparts. In this paper, we argue that the performance of propose-and-rank models are underestimated due to the predefined manners: 1) Hand-designed rules are hard to guarantee the complete coverage of targeted segments. 2) Densely sampled candidate moments cause redundant computation and degrade the performance of ranking process. To this end, we propose a novel model termed LPNet (Learnable Proposal Network for NLVL) with a fixed set of learnable moment proposals. The position and length of these proposals are dynamically adjusted during training process. Moreover, a boundary-aware loss has been proposed to leverage frame-level information and further improve performance. Extensive ablations on two challenging NLVL benchmarks have demonstrated the effectiveness of LPNet over existing state-of-the-art methods.

## introduction natural language video localization (nlvl), aka, video grounding or video moment localization, has got unprecedented attention in both cv and nlp communities @xcite . as shown in figure 1 , nlvl aims to localize the video segment relevant to the query by locating the start and end timestamps in an untrimmed video. it is challenging since it needs to not only understand the video and the sentence content but also find out the precise temporal boundaries. moreover, nlvl is helpful to numerous downstream video understanding tasks, e.g., content retrieval @xcite , relation detection @xcite , and vqa @xcite . currently, state-of-the-art nlvl methods can be roughly grouped into two categories according to how the video segments are detected, namely propose-and-rank and proposal-free methods: the idea of the propose-and-rank approach @xcite @xcite @xcite @xcite is intuitive, which follows the same spirits of anchor-based object detectors, e.g., faster r-cnn @xcite . this kind of methods firstly defines a series of manuallydesigned temporal bounding boxes as moment proposals. then, they match each candidate with the sentence in a common feature space and compute matching scores for all the candidates. thus the localization problem is reformulated into a ranking problem. however, these methods suffer from two inherent drawbacks due to the predefined manners: 1) even though they elaborately design a series of hyperparameters (e.g., temporal scales and sample rates), these hand-designed rules are hard to guarantee the complete coverage of targeted video segments, and consequently tend to produce inaccurate boundaries. 2) a vast number of proposals are required to achieve high recall, which causes redundant computation and degrades the results of ranking process. another type of solution proposal-free approach @xcite @xcite @xcite ) mitigates these defects. instead of predefining a series of temporal proposals, they directly predict the start and the end boundaries or regress the locations of the query-related video segments. benefit from such design, proposal-free methods get rid of placing superfluous temporal anchors, i.e., they are more computation-efficient. furthermore, without fixing the position and length of the moment proposals, these methods are flexible to adapt to video segments with diverse lengths. compared to propose-and-rank methods, there are two main limitations of proposal-free methods @xcite : 1) they overlook the rich information between start and end boundaries because they are hard to model the segment-level interaction. 2) they always suffer from severe imbalance between the positive and negative training samples. up to now, almost all the propose-and-rank methods have inferior performance. we argue that the performance of the propose-and-rank methods are underestimated due to current predefined designs. in this paper, we propose a novel propose-and-rank model with learnable moment proposals, termed lpnet. without fixed dense proposals, only a sparse set of proposals are required to obtain decent performance. in addition, there is no need to worry about the design of hyper-parameters because it is adaptable to targeted segments with diverse positions and lengths. obviously, as a propose-andrank method, lpnet also avoids the defects of the proposal-free approach. specifically, lpnet places a fixed set of learnable temporal proposals represented by 2-d coordinates indicating the centers and lengths of video segments. these proposals are used to extract visual features of moment of interest (moi). in order to model the relative relations among candidates, a module has been proposed to make the candidates interact with each other using the selfattention mechanism. then an individual classifier is used to predict the matching score between these proposals and the sentence query. during the training process, the coordinates of the proposal with maximum score are adjusted by a dynamic adjustor at each iteration. after sufficient iterations, these learned moment proposals will statistically represent the prior distributions of ground-truth segments on the dataset. in addition, we empirically find that the propose-and-rank models always obtain sub-optimal results without frame-level supervision. a boundary-aware predictor has been proposed to regularize the model to utilize framelevel information, which further boosts the grounding performance. we demonstrate the effectiveness of our model on two challenging nlvl benchmarks (charades-sta, and activitynet captions) by extensive ablative studies. particularly, our model achieves new state-of-the-art performance over all datasets and evaluation metrics. the propose-and-rank approaches @xcite @xcite @xcite solve the nlvl task by matching the predefined video moment proposals (e.g., in sliding window manner) with the language query and choose the best matching video segment as the final result. @xcite proposed a ctrl model. it takes video moments predefined through sliding windows as input and jointly models text query and video clips, then outputs alignment scores and action boundary regression results for candidate clips. @xcite proposed mcn which effectively localizes language queries in videos by integrating local and global video features over time. to improve the performance of the propose-and-rank method, some works devote to improve the quality of the proposals. @xcite injected text features early on when generating clip proposals to eliminate unlikely clips and thus speed up processing and boost performance. @xcite proposed to explicitly model moment-wise temporal relations as a structured graph and devised an iterative graph adjustment network to jointly learn the best structure in an end-to-end manner. the others mainly worked on designing a more effective multi-modal interaction network. @xcite utilized a query the lady spins the stick using her neck. 3d conv glove cross-modal attention boundary-aware predictor feature encoder feature encoder temporal roi align interactive rating layer 0.1 0.8 0.2 learnable proposals dynamic adjustor feature extractor video feature encoder boundary-aware predictor propose-and-rank module start end figure 2 : the architecture of lpnet for nlvl. feature extractor transforms input video and language query into feature space. feature encoder further refines video and language feature, and produces the multi-modal feature. a series of learnable proposal boxes are proposed which can be updated by dynamic adjustor during training. interactive rating layer scores each candidate generated by proposal boxes and the candidate with highest score is the final prediction. boundary-aware predictor takes multi-modal feature as input and predict the distribution of start and end timestamps, which is an auxiliary task to regularize the model to get better performance. there are also some other works @xcite solving the nlvl task by rl, which formulates the selection of start and end timestamps as a sequential decision making process. and some concurrent nlvl works also borrow the design of visual transformers @xcite to explore transformer-based nlvl model @xcite . end-to-end object detection. the development of nlvl is inspired by the success of object detection methods. object detection aims to obtain a tight bounding box and a class label for each object. it can be categorized into anchor-based and anchor-free approaches. traditional anchor-based models @xcite have dominated this area for many years, which place a series of anchor boxes uniformly and do the classification and regression to determine the position and class for the objects. anchor-free models @xcite are becoming prosperous, which have been promoted by the development of key point detection. they directly predict key points and group them together to determine objects. recently, end-to-end object detectors based on sparse candidates have drawn large amount of attentions. detr @xcite utilizes a sparse set of object queries to interact with the global feature. benefit from the bipartite matching predictions and ground-truths, detr can discard add&layernorm multi-head attention convolution feed forward add&layernorm add&layernorm add&layernorm add&layernorm addition q v × × × row/column softmax similarity matrix cross-modal fusion v q × × figure 3: the overview of the feature encoder. the embedding encoder consists of several conv-layers, a self-attention layer and a feed-forward layer together with layer normalization and residual connection. the cross-modal attention layer learns the multi-modal interaction and outputs the query-guided video feature. the nms procedure while achieving remarkable performance. similarly, sparse r-cnn @xcite provides a fixed set of sparse learnable object proposals and performs classification and localization. our model borrows the similar idea from sparse r-cnn. ## approach we formally define nlvl task as follows. given an untrimmed video as @xmath0, where t and m are the number of video frames and query words, nlvl needs to predict the start and end timestamps (t s , t e ) of the video segment described by the language query q. for each video, we extract its visual features @xmath1. for each query, we initialize the word features @xmath2. the overall architecture of the proposed lpnet is shown in figure 2 . in this section, we first introduce each component of lpnet in sequence. then, we show the training and inference stage in details. ## ablation studies in this section, we conduct ablative experiments with different variants to better investigate our approach. ## conclusions in this paper, we present a novel propose-and-rank model with learnable moment proposals for nlvl. compared to the existing propose-and-rank method with predefined temporal boxes, our model improves the performance significantly because 1) our model disengages from the hand-designed rules for bounding boxes so that it can produce more accurate temporal boundaries; 2) sparse sampled candidate release the pressure for subsequent ranking process; 3) boundary-aware loss regularize the model to avoid sub-optimum. in the future, we are going to explore a more effective way to learn better proposals and extend this idea to other tasks.