Adaptive Proposal Generation Network for Temporal Sentence Localization in Videos (2021)

Abstract: We address the problem of temporal sentence localization in videos (TSLV). Traditional methods follow a top-down framework which localizes the target segment with pre-defined segment proposals. Although they have achieved decent performance, the proposals are handcrafted and redundant. Recently, bottom-up framework attracts increasing attention due to its superior efficiency. It directly predicts the probabilities for each frame as a boundary. However, the performance of bottom-up model is inferior to the top-down counterpart as it fails to exploit the segment-level interaction. In this paper, we propose an Adaptive Proposal Generation Network (APGN) to maintain the segment-level interaction while speeding up the efficiency. Specifically, we first perform a foreground-background classification upon the video and regress on the foreground frames to adaptively generate proposals. In this way, the handcrafted proposal design is discarded and the redundant proposals are decreased. Then, a proposal consolidation module is further developed to enhance the semantics of the generated proposals. Finally, we locate the target moments with these generated proposals following the top-down framework. Extensive experiments show that our proposed APGN significantly outperforms previous state-of-the-art methods on three challenging benchmarks.

## introduction temporal sentence localization in videos is an important yet challenging task in natural language processing, which has drawn increasing attention over the last few years due to its vast potential applications in information retrieval @xcite and human-computer interaction @xcite . it aims to ground the most relevant video segment according to a given frame-wise probabilities video-query interacted representation start end figure 1: (a) an example of temporal sentence localization in videos. (b) the top-down framework predicts the confidence scores of a large number of pre-defined proposals for ranking. (c) the bottom-up framework regresses the probabilities of all frames as start or end boundaries. to avoid the inherent drawbacks of proposal design in the top-down framework and maintain the localization performance, in this paper, we propose an adaptive proposal generation network (apgn) for an efficient and effective localization approach. firstly, we perform boundary regression on the foreground frames to generate proposals, where foreground frames are obtained by a foregroundbackground classification on the entire video. in this way, the noisy responses on the background frames are attenuated, and the generated proposals are more adaptive and discriminative compared to the pre-defined ones. secondly, we perform proposal ranking to select target segment in a top-down manner upon these generative proposals. as the number of proposals is much fewer than the predefined methods, the ranking stage is more efficient. furthermore, we additionally consider the proposalwise relations to distinguish their fine-grained semantic details before the proposal ranking stage. to achieve the above framework, apgn first generates query-guided video representations after encoding video and query features and then predicts the foreground frames using a binary classification module. subsequently, a regression module is utilized to generate a proposal on each foreground frame by regressing the distances from itself to start and end segment boundaries. after that, each generated proposal contains independent coarse semantic. to capture higher-level interactions among proposals, we encode proposal-wise features by incorporating both positional and semantic information, and represent these proposals as nodes to construct a proposal graph for reasoning correlations among them. consequently, each updated proposal obtains more fine-grained details for following boundary refinement process. our contributions are summarized as follows: ## related work temporal sentence localization in videos is a new task introduced recently @xcite , which aims to localize the most relevant video segment from a video with sentence descriptions. various algorithms @xcite @xcite @xcite @xcite have been proposed within the topdown framework, which samples candidate segments from a video first, then integrates the sentence representation with those video segments individually and evaluates their matching relationships. some of them @xcite propose to use the sliding windows as proposals and then perform a comparison between each proposal and the input query in a joint multi-modal embedding space. to improve the quality of the proposals, @xcite pre-cut the video on each frame by multiple pre-defined temporal scale, and directly integrate sentence information with fine-grained video clip for scoring. @xcite further build a 2d temporal map to construct all possible segment candidates by treating each frame as the start or end boundary, and match their semantics with the query information. although these methods achieve great performance, they are severely limited by the heavy computation on proposal matching/ranking, and sensitive to the quality of pre-defined proposals. recently, many methods @xcite @xcite @xcite @xcite propose to utilize the bottom-up framework to overcome above drawbacks. they do not rely on the segment proposals and directly select the starting and ending frames by leveraging cross-modal interactions between video and query. specifically, they predict two probabilities at each frame, which indicate whether this frame is a start or end frame of the ground truth video segment. although these methods perform segment localization more efficiently, they lose the segment-level interaction, and the redundant regression on background frames may provide disturbing noise for boundary decision, leading to worse localization performance than top-down methods. in this paper, we propose to preserve the segment-level interaction while speeding up the query encoder video-query interaction video encoder (a) query-guided video feature generation (b) proposal generation (c) proposal consolidation (d) localization foreground-background classification predict mask boundary regression regress start end proposal encoder extract ... localization efficiency. specifically, we design a binary classification module on the entire video to filter out the background responses, which helps model focus more on the discriminative frames. at the same time, we replace the pre-defined proposals with the generated ones and utilize a proposal graph for refinement. ## conclusion in this paper, we introduce apgn, a new method for temporal sentence localization in videos. our core idea is to adaptively generates discriminative proposals and achieve both effective and efficient localization. that is, we first introduce binary classification before the boundary regression to distinguish the background frames, which helps to filter out the corresponding noisy responses. then, the regressed boundaries on the predicted foreground frames are taken as segment proposals, which decreases a large number of poor quality proposals compared to the pre-defined ones in top-down framework. we further learn higher-level feature interactions between the generated proposals for refinement via a graph convolutional network. our framework achieves state-of-the-art performance on three challenging benchmarks, demonstrating the effectiveness of our proposed apgn.