Learning to Focus on the Foreground for Temporal Sentence Grounding (2022)

Abstract: Temporal sentence grounding (TSG) is crucial and fundamental for video understanding. Previous works typically model the target activity referred to the sentence query in a video by extracting the appearance information from each whole frame. However, these methods fail to distinguish visually similar background noise and capture subtle details of small objects. Although a few recent works additionally adopt a detection model to filter out the background contents and capture local appearances of foreground objects, they rely on the quality of the detection model and suffer from the time-consuming detection process. To this end, we propose a novel detection-free framework for TSG—Grounding with Learnable Foreground (GLF), which efficiently learns to locate the foreground regions related to the query in consecutive frames for better modelling the target activity. Specifically, we first split each video frame into multiple patch candidates of equal size, and reformulate the foreground detection problem as a patch localization task. Then, we develop a self-supervised coarse-to-fine paradigm to learn to locate the most query-relevant patch in each frame and aggregate them among the video for final grounding. Further, we employ a multi-scale patch reasoning strategy to capture more fine-grained foreground information. Extensive experiments on three challenging datasets (Charades-STA, TACoS, ActivityNet) show that the proposed GLF outperforms state-of-the-art methods.

## introduction temporal sentence grounding (tsg) @xcite is an important yet challenging topic of video understanding in computer vision. given an untrimmed video, it aims to retrieve a temporal segment that semantically corresponds to a given sentence query, as shown in figure 1 @xcite . compared to other videoand-language tasks like video captioning @xcite and video action localization @xcite @xcite , tsg is substantially more challenging as it need not only capture the complicated visual and textual information, but also learn the complex multi-modal interactions among them for modelling the target activity. to localize the target segment, most previous works either pre-define abundant segment proposals @xcite @xcite @xcite @xcite to match the query semantic for ranking and selection, or employ proposal-free frameworks @xcite @xcite to directly regress the start/end timestamps of the segment. although these methods have made significant progress in recent years, they extract the appearance information of each whole frame among the entire video, thus limiting the effective integration of the foreground contexts for modelling the target activity due to the visually similar backgrounds and the missing subtle details of small objects. to alleviate such limitations, a few recent works @xcite attempt to additionally adopt a pre-processing detection model (i.e., faster r-cnn @xcite ) that detects the foreground objects for filtering out the background noise. however, they rely on the quality of the detection model while suffering from the time-consuming detection process. based on the above considerations, this paper aims to develop a detection-free grounding network, which efficiently selects the most queryrelevant region in each frame to represent the framelevel features among the entire video for better modelling the target activity. as illustrated in figure 1(b), in order to effectively represent different local regions in each frame, we divide it into multiple patches that serve as the region candidates to be selected according to their semantic similarity with the query. once the best patch is determined in each frame during the network learning, they are extracted to model the activity by spatial-temporal correlation reasoning. compared to previous methods, such detection-free network provides more fine-grained foreground details by filtering out the background regions and capturing the local contexts in an efficient and end-to-end manner, leading to better grounding results. to this end, we propose a novel tsg model, called grounding with learnable foreground (glf), which learns to focus on the query-relevant foreground regions among video frames to model the fine-grained target activity for more accurate grounding. specifically, we reformulate the foreground detection problem as a patch localization task. considering the spatial-temporal information within the video, we extract 3d spatial-temporal patches instead of 2d spatial ones on the video clips (i.e., several consecutive frames). we first introduce a 3d patch embedding layer to encode the local information of each patch candidate, and concatenate it with an additional global representation extracted from its current video clip. then, we interact the patch candidates in each clip with the query semantic to learn their matching scores for distinguishing the foreground and background patches. particularly, we develop a two-level coarse-to-fine paradigm to gradually localize the most relevant (foreground) patch in each clip. at last, we aggregate the representations of the most relevant patches among the entire video to model the target activity. in addition, considering the sizes of the foreground regions may vary in different videos, we further extend the glf model with multi-scale patch design to capture more finegrained and complete foreground information for better grounding. our main contributions are summarized as follows: ## related work temporal sentence grounding (tsg) is a new task introduced recently @xcite , which aims to localize the most relevant video segment from a video with sentence descriptions. most previous works @xcite @xcite @xcite @xcite generate multiple segment proposals and then rank them according to the similarity between proposals and the query for selecting the best matching one. instead of generating complex proposals, some works @xcite @xcite directly regress the temporal locations of the target bi-gru glove c3d 3d patch embedding global feature combine positional embedding transformer transformer multi-modal interaction scoring module weighting module coarse-level fine-level grounding heads | | | multi-modal encoders self-supervised patch localization q v p q p p match ground 1 2 5 5 p n p selection policy gauss distribution figure 2: overview of the proposed glf model. it consists of the multi-modal encoders and the self-supervised patch localization. ## our method given an untrimmed video v and a sentence query q, the tsg task aims to determine the start and end timestamps of a specific video segment referring to the sentence query. formally, we represent the video as @xmath0, where c t is the t-th clip and t is the total clip number. we also denote the query as @xmath1, where s is the length of the sentence. in this section, we introduce the overall architecture of our proposed glf model. as shown in figure 2 , the model consists of two main parts: multi-modal encoders and self-supervised patch localization. first of all, glf splits each video clip into multiple spatial-temporal patch candidates of equal size and encodes them with a shared 3d embedding layer. then, the model extracts the query embeddings and interact them with all patch candidates. after that, a coarse-to-fine patch localization paradigm is proposed to gradually score the patches in each clip according to their query-relevant similarity. at last, a single best patch in each clip is selected by a learned policy module to represent the current query-guided clip feature for final grounding. considering the sizes of the foreground regions may vary in different videos, we further extend the model with a multi-scale patch design to aggregate different-level foreground contexts. we elaborate on each module below. ## conclusion in this paper, we make the first attempt to propose a novel detection-free framework for temporal sentence grounding (tsg), called grounding with learnable foreground (glf). in particular, we split each video frame into patches with multiple scales, and reformulate the foreground detection problem as a patch localization task. in detail, we interact each patch with the query semantic to learn their matching scores supervised by our newly designed self-supervised losses. further, we develop a two-level coarse-to-fine paradigm to gradually localize the most query-relevant (foreground) patch in each clip. moreover, considering the sizes of the foreground regions may vary in different videos, we extend the glf model with multi-scale patch design to capture more fine-grained and complete foreground information for better grounding. experimental results on three challenging datasets (charades-sta, tacos, activitynet) validate the effectiveness of our proposed model.