Video-Text Prompting for Weakly Supervised Spatio-Temporal Video Grounding (2024)

Abstract: Weakly-supervised Spatio-Temporal Video Grounding(STVG) aims to localize target object tube given a text query, without densely annotated training data. Existing methods extract each candidate tube feature independently by cropping objects from video frame feature, discarding all contextual information such as position change and inter-entity relationship. In this paper, we propose Video-Text Prompting(VTP) to construct candidate feature. Instead of cropping tube region from feature map, we draw visual markers(e.g. red circle) over objects tubes as video prompts; corresponding text prompt(e.g. in red circle) is also inserted after the subject word of query text to highlight its presence. Nevertheless, each candidate feature may look similar without cropping. To address this, we further propose Contrastive VTP(CVTP) by introducing negative contrastive samples whose candidate object is erased instead of being highlighted; by comparing the difference between VTP candidate and the contrastive sample, the gap of matching score between correct candidate and the rest is enlarged. Extensive experiments and ablations are conducted on several STVG datasets and our results surpass existing weakly-supervised methods by a great margin, demonstrating the effectiveness of our proposed methods.

## introduction the task of stvg is of high importance to real world applications such as general artificial intelligence for video understanding, information retrieval for surveillance systems and humanmachine interaction, etc. to relieve the reliance on heavily annotated training data, weakly-supervised stvg deserves more research attention. regrettably, training under this setting is an extremely challenging because only the pairing information between video clips and their corresponding query * the authors contributed equally to this work. time crop (a). existing paradigms. (b). video-text prompting. matching figure 1: comparison between existing paradigms (chen et al., 2019b; li et al., 2023) and the proposed video-text prompting(vtp). given pre-extracted candidate tube boxes, (a) existing methods obtain candidate feature by cropping from frame feature, resulting in contextual information loss; (b) our proposed vtp constructs context-preserving candidates by adding visual and textual prompts to the input. it is common for weakly-supervised methods to reformulate the grounding problem as a retrieval task where candidate tube boxes are obtained with pre-trained object detectors and trackers then later ranked based on the query to find the best pick. existing methods reason with the entity's tube feature cropped from the global frame feature given its boxes obtained in pre-processing step, discarding all contextual information such as the entity's relationship with other entities, its moving trajectory and so on(illustrated in fig. 1 (a) ). we note that this is an inherent difficulty for existing methods as their feature extraction is usually done together with the pre-processing step. to overcome this weakness, we propose video-text prompting(vtp). in detail, by transforming the tube boxes obtained in pre-processing into video prompt(e.g. red circles) that are drawn as markers on the input video frames, as shown in fig. 1 (b) ; we are able to preserve all useful context information for reasoning. meanwhile we prompt the query sentence correspondingly by inserting text prompt(e.g. in red circle) after the subject word. we refer to candidate instances created in this way as video prompted(vp) candidates. notably, this will create a visual discrepancy with the prompted query if the visual prompt lands on an incorrect candidate thus lowering its matching potential. nevertheless, this discrepancy can be subtle as it may contradict with the prompted query partially. for example, the vp candidate 2 in fig. 2 is an interference as it partially matches the prompted query: an adult in red circle? (yes), he is walking a dog? (no). to address this issue, we further propose contrastive vtp(cvtp) where we construct a contrastive video prompted(cvp) counterpart for each vp candidate by erasing its presence rather than highlighting it. intuitively, the matching score of the cvp counterpart will be decimated in case the correct candidate is erased and meanwhile it is high for those incorrect candidate who is irrelevant to the query text. thus by calculating the matching score difference between vp candidates and its cvp counterparts, we suppress the interference candidates and enlarge the gap between the correct candidate and the incorrect ones, as shown in fig. 2 . another challenge in weakly-supervised stvg is the temporal reasoning required to refine the selected candidate tube along the time axis. we address this by implementing a light-weight encoderdecoder transformer where the self-attention layers in encoder models temporal interaction between the prompted video frames and the cross-attention layers in decoder models the multi-modal reasoning between visual and linguistic feature. we conduct extensive experiments on two commonly used stvg datasets and our results surpasses existing weakly-supervised counterparts by a notable margin, which demonstrates its effectiveness. we list our contributions as follows: visual prompting originated for nlp community, prompting methodology can be generalized as adding fixed or trainable parameters to the raw input. most of the early prompting inspired works for visual-related tasks @xcite @xcite only prompt text or class labels, there is also works explore visual prompts by adding learnable pixels or tokens to raw image input @xcite @xcite . the form of visual prompts become diverse in recent years, such as bounding boxes @xcite , masks @xcite or even mouse clicks @xcite . although the different prompt types, these work treat visual prompts as a visual prior or anchor to better understand the prompted region. in contrast, we use visual prompts as an local emphasis to contrast with other entities. the most similar work to our setup is @xcite , which also uses red circle as visual prompts to highlight the region for fine-grained local perception. however, our proposed cvtp not just employs visual prompts, it further leverages corresponding prompted texts and cvp counterpart to empower the contrast between candidates. weakly supervised video grounding fullysupervised stvg methods @xcite @xcite @xcite hold sota performance with a large margin compared with weakly-supervised counterparts; nevertheless, the requirements of framelevel bounding box annotation and temporal boundary with second-level precision is impractical when the model needs to be trained for a new application with different data distributions. however, research