Generating Structured Pseudo Labels for Noise-resistant Zero-shot Video Sentence Localization (2023)

Abstract: Video sentence localization aims to locate moments in an unstructured video according to a given natural language query. A main challenge is the expensive annotation costs and the annotation bias. In this work, we study video sentence localization in a zero-shot setting, which learns with only video data without any annotation. Existing zero-shot pipelines usually generate event proposals and then generate a pseudo query for each event proposal. However, their event proposals are obtained via visual feature clustering, which is query-independent and inaccurate; and the pseudo-queries are short or less interpretable. Moreover, existing approaches ignores the risk of pseudo-label noise when leveraging them in training. To address the above problems, we propose a Structure-based Pseudo Label generation (SPL), which first generate free-form interpretable pseudo queries before constructing query-dependent event proposals by modeling the event temporal structure. To mitigate the effect of pseudo-label noise, we propose a noise-resistant iterative method that repeatedly re-weight the training sample based on noise estimation to train a grounding model and correct pseudo labels. Experiments on the ActivityNet Captions and Charades-STA datasets demonstrate the advantages of our approach. Code can be found at https://github.com/minghangz/SPL .

## introduction video sentence localization, which aims to localize the most salient video segments from an untrimmed video given a free-form nature language query, has attained increasing attention due to its potential applications in video surveillance @xcite , robot manipulation @xcite , etc. the free-form natural language queries allow the model to be flexibly adapted to the requirements of different practical applications. step 1. pseudo query: a man cleans the snow with a brush similarity time step 3. pseudo event: [4s, 15s] step 1. pseudo event: [8s, 11s] aligned query: a man cleans the snow on the car gt : … … fully supervised weakly supervised video: [3s, 14.5s] query: a man cleans the snow on the car … … video: (a) (b) step 2. similarity: figure 1: (a) training data for fully-supervised models. (b) training data for weakly-supervised models. (c) the zero-shot models are trained with videos only. existing pipeline may generate unaligned pseudo event-query pairs. (d) we construct query-dependent event proposals by modeling the event temporal structure. ## approach the overview of our model design is illustrated in figure 2 . our method is divided into four steps. in the first step, we generate pseudo queries for a given video. to obtain realistic free-form nature language queries, we sample video frames and generate captions using a pre-trained image caption model, which will serve as our pseudo query candidates. in the second step, we generate pseudoevent proposals for each pseudo query. as the events described by the query should have a certain structure, i.e. the relevance to the query in the event should be high, while the relevance to the query outside the event should be low, we calculate the similarity between each query candidate and each video frame. then, for each pseudo query, we select the best event proposal with the largest gap between the correlation within the event and the correlation outside the event, and the gap is served as the quality of that pseudo query. as a good pseudo query should not be too general (e.g. 'there is a person' is a bad query), it should be significantly more relevant to the corresponding event in the video than other video segments. thus, in the third step, we will only keep the top-k highquality proposal-query pairs. in the last step, we will train a fully supervised model using the filtered pseudo-query-event pair. to reduce the noise in the pseudo-labels, we propose to estimate the noise and then re-weight each sample, while refining the pseudo-labels during training. ## experiments to evaluate our method, we conduct experiments on the charades-sta @xcite and activi-tynet captions @xcite dataset.