Rethinking the Video Sampling and Reasoning Strategies for Temporal Sentence Grounding (2022)

Abstract: Temporal sentence grounding (TSG) aims to identify the temporal boundary of a specific segment from an untrimmed video by a sentence query. All existing works first utilize a sparse sampling strategy to extract a fixed number of video frames and then interact them with query for reasoning.However, we argue that these methods have overlooked two indispensable issues:1) Boundary-bias: The annotated target segment generally refers to two specific frames as corresponding start and end timestamps. The video downsampling process may lose these two frames and take the adjacent irrelevant frames as new boundaries.2) Reasoning-bias: Such incorrect new boundary frames also lead to the reasoning bias during frame-query interaction, reducing the generalization ability of model.To alleviate above limitations, in this paper, we propose a novel Siamese Sampling and Reasoning Network (SSRN) for TSG, which introduces a siamese sampling mechanism to generate additional contextual frames to enrich and refine the new boundaries. Specifically, a reasoning strategy is developed to learn the inter-relationship among these frames and generate soft labels on boundaries for more accurate frame-query reasoning. Such mechanism is also able to supplement the absent consecutive visual semantics to the sampled sparse frames for fine-grained activity understanding.Extensive experiments demonstrate the effectiveness of SSRN on three challenging datasets.

## introduction temporal sentence grounding (tsg) is an important yet challenging task in natural language pro- * equal contributions. † corresponding author. original segment ... ... (b) an illustration of the video feature sampling of existing tsg methods. video original segment ... ... (b) an illustration of the video feature sampling of existing tsg methods. (c) an overview of our siamese sampling strategy. (c) an overview of our siamese sampling strategy. new segment new segment refine refine siamese sampling siamese sampling original segment original segment contextual frames contextual frames enrich enrich contextual frames contextual frames enrich enrich sparse sampling sparse sampling ... ... video video ... ... ... ... ... ... new segment new segment refined segment refined segment figure 1: (a) an example of temporal sentence grounding task. (b) all existing tsg methods generally utilize a downsampling process to evenly extract a fixed number of frames from a long video. however, the new target segment is obtained by rounding operation and may introduces boundary bias since some original boundary frames are lost. (c) we propose a siamese sampling strategy to extract additional adjacent frames to enrich and refine the information of the sampled frames for generating more accurate boundary of the new segment. previous tsg methods @xcite @xcite @xcite @xcite @xcite generally follow an encoding-then-interaction framework that first extracts both video and query features and then conduct multi-modal interactions for reasoning. since many videos are overlong while corresponding target segments are short, these methods simply utilize a sparse sampling strategy shown in figure1 (b), which samples a fixed number of frames from each video to reconstruct a shorter video, and then learn frame-query relations for segment inferring. we argue that existing learning paradigm suffers from two obvious limitations: 1) boundary-bias: each video has a query-related segment, which refers to two specific frames as its start and end timestamps. traditional sparse downsampling strategy extracts frames from videos with a fixed interval. a rounding operation is then applied to map the annotated segment to the sampled frames by keeping the same proportional length in both original and new videos. as a result, the groundtruth boundary frames may be filtered out and the query-irrelevant frames will be regarded as the actual boundaries, generating wrong labels for latter training. 2) reasoning-bias: the query-irrelevant boundary frames in the newly reconstructed segment will also lead to incorrect frame-query interaction and reasoning in the training process, reducing the generalization ability of model. to alleviate these two issues, a straightforward idea is to filter out the sampled boundary frames in the new segment if they are query-irrelevant. however, this will destroy the true segment length when we transfer the downsampled segment back to the original one during the inference process. another straightforward idea is to directly keep the appropriate segment length (by float values) in the newly reconstructed video and then reason the query content in the new boundary to determine what percentage of this boundary is correct. however, the queryirrelevant boundaries lack sufficient query-related information for boundary reasoning. based on the above considerations, we aim to extract additional frames adjacent to the sampled frames to enrich and refine their information for supplementing the consecutive visual semantics. in this way, the new boundary frames are well semantic-correlated to its original adjacent boundaries. based on the refined boundary frames, we can keep and learn the appropriate segment length of the downsampled video for query reasoning. moreover, other inner frames are also enriched by their neighbors, captur-ing more consecutive visual appearances for fully understanding the entire activity. therefore, in this paper, we propose a novel siamese sampling and reasoning network (ssrn) for temporal sentence grounding task to generate additional contextual frames to enrich and refine the new boundaries. specifically, we treat the sparse sampled video frames as anchor frames, and additionally extract several frames adjacent to each anchor frame as the siamese frames for semantic sharing and enriching. a siamese knowledge aggregation module is designed to explore internal relationships and aggregate contextual information among these frames. then, a siamese reasoning module supplements the fine-grained contexts of siamese frames into the anchor frames for enriching their semantics. in this way, the query-related information are added into the new boundaries thus we can utilize an appropriate float value to represent the new segment length for query reasoning, addressing both boundary-and reasoning-bias. moreover, other sampled frames are also equipped with more consecutive visual semantics from their original neighbors, which further benefits more fine-grained learning process. our contributions are summarized as follows: ## related work temporal sentence grounding (tsg) is a new task introduced recently @xcite , which aims to localize the most relevant video segment from a video with sentence descriptions. all existing methods follow an encoding-then-interaction framework that first extracts video/query features and then conduct multi-modal interactions for segment inferring. based on the interacted multi-modal features, traditional methods follow a propose-and-rank paradigm to make predictions. most of them @xcite @xcite @xcite @xcite @xcite typically utilize a proposal-based grounding head that first generates multiple candidate segments as proposals, and then ranks them according to their similarity with the query semantic to select the best matching one. some of them @xcite directly utilize multi-scale sliding windows to produce the proposals and subsequently integrate the query with segment representations via a matrix operation. to improve the quality of the proposals, latest works @xcite @xcite @xcite @xcite ) integrate sentence information with each fine-grained video clip unit, and predict the scores of candidate segments by gradually merging the fusion feature sequence over time. recently, some proposal-free works @xcite @xcite @xcite @xcite @xcite directly predict the temporal locations of the target segment without generating complex proposals. these works directly select the starting and ending frames by leveraging cross-modal interactions between video and query. specifically, they either regress the start/end timestamps based on the entire video representation @xcite , or predict at each frame to determine whether this frame is a start or end boundary @xcite @xcite @xcite . although the above two types of methods have achieved great performances, their video sampling strategy in encoding part is unreasonable that can lead to both boundary and reasoning bias. specifically, the boundary bias is defined as the incorrect boundary of the new segment reconstructed by the video sparse sampling. the reasoning bias is defined as the incorrect correlation learning between the query-irrelevant frames and query. in this paper, we aim to reduce the above bias by proposing a new siamese sampling and reasoning strategy to enrich the sampled frames and further refine the reconstructed segment boundary. ## the proposed method given an untrimmed video and a sentence query, we represent the video as v with a frame number of t . similarly, the query with n words is denoted as q. temporal sentence grounding (tsg) aims to localize a segment (τ s , τ e ) starting at timestamp τ s and ending at timestamp τ e in video v, which corresponds to the same semantic as query q. the overall architecture of the proposed siamese sampling and reasoning network (ssrn) method is illustrated in figure 2 . the ssrn framework contains four main components: (1) siamese sampling and encoding: we sparsely downsample each long video into the anchor frames, and a new siamese sampling strategy additionally samples their adjacent frames as siamese frames. a video/query encoder then extracts visual/query features from all sampled video frames and query sentence respectively. (2) multi-modal interaction: after that, we interact the query features with the visual features for cross-modal interaction. (3) multimodal reasoning: next, to supplement the knowledge of siamese frames into the anchor frames, a siamese knowledge aggregation module is developed to determine how much the information of siamese frames are needed to inject into the anchor ones. then, a reasoning module is utilized to enrich the anchor frames with the aggregated semantic knowledge. in this way, the contexts of both new boundaries and other sparse frames are enriched and can better represent the full and consecutive visual semantics. (4) grounding heads with soft labels: at last, we employ the grounding heads with soft label to predict more accurate boundaries via float value to keep the appropriate segment length. we illustrate the details of each component in the following subsections. ## conclusion in this paper, we propose a novel siamese sampling and reasoning network (ssrn) to alleviate the limitations of both boundary-bias and reasoningbias in existing tsg methods. in addition to the original anchor frames, our model also samples a certain number of siamese frames from the same video to enrich and refine the visual semantics of the anchor frames. a soft label is further exploited to supervise the enhanced anchor features for predicting more accurate segment boundaries. experimental results show both effectiveness and efficiency of our ssrn on three challenging datasets.